Blackbox Assessment
The English user guide is currently in beta preview. Most of the documents have been automatically translated from the Japanese version. Should you find any inaccuracies, please reach out to Flatt Security.
Overview
Takumi's blackbox assessment feature receives an application's URL and authentication information, performs simulated attacks on the application, and outputs assessment results as a report on the web.
It can be used from the Shisho Cloud byGMO web interface.
Regarding Credit Consumption
Credits are required to use this feature.
You must set a credit threshold when starting an assessment, and scans are performed within that limit. When the credit threshold is reached during a scan, the scan stops after the currently running test completes.
Even if actual credit consumption exceeds the configured threshold, the excess credits will not be charged.
For example, if the credit threshold is set to 10 and the actual consumption is 11, the final billed credit consumption will be 10.
Organization or Target Ownership Verification
Before starting an assessment, you need to verify your organization or prove ownership of the target application for blackbox assessment. For details, please refer to "Pre-Assessment Organization Verification or Ownership Verification".
How to Start an Assessment
Click "Assessments" in the sidebar, then click the "Create Assessment" button in the upper right corner of the screen to start an assessment.
Basic Settings
- Enter a name to identify the assessment in "Assessment Name".
- Select the report language (English or Japanese) in "Report Language".
- Select "Web App URLs" as the assessment target.
Assessment Type
You can select one of the following two modes:
- Full Assessment mode: Crawling and the initial scan of the target application are performed in one seamless process. After completion, the assessment pauses in a "Pending" state. See Reviewing Results and Running Additional Scans for next steps.
- Scoped Assessment mode: Takumi stops at the end of crawling. After that, you set priorities for the features and perspectives you want to assess, as well as the credit threshold for the scan, then resume the assessment. This is recommended for those who want fine-grained control over the assessment scope.
Credit Threshold
Credit thresholds can be set separately for crawling and scanning (for "Scoped Assessment" mode, only crawling). Takumi performs crawling and scanning within the specified credit thresholds.
Web Application Settings
- Enter the URL of the target application in "Application URL".
- Configure the authentication credentials for logging into the application in "Authentication Information".
Reviewing Crawl Results and Running Additional Crawls
Setting Priorities and Credit Threshold
If you select "Scoped Assessment" mode, the assessment pauses after crawling completes. Opening the assessment page displays a matrix of detected features and assessment perspectives.
On this screen, you can set the credit threshold for scanning and assessment priorities to start the assessment.
You can set priorities of "Auto, High, Medium, Low, None" for each combination of feature and perspective. Higher-priority items are scanned first. When "Auto" is selected, Takumi automatically determines the priority based on risk analysis.
Once priorities and the credit threshold are configured, click the "Start Pentesting" button to begin the scan.
Additional Crawl
When an assessment is in the "Crawled" state, you can select "Additional Crawl" from the menu to run additional crawling. This is useful when you want to discover endpoints that were not found during the initial crawl.
The following settings are available for additional crawls:
- Credit Threshold: The maximum credit limit for the additional crawl
- Additional Crawl Instructions: Free-form instructions to guide the crawl toward specific areas (e.g., focusing on specific API paths)
When the additional crawl completes, newly discovered endpoints are added to the matrix.
Running an additional crawl does not guarantee that new endpoints will be discovered. Credits are consumed even if no new endpoints are found.
Manual Editing of Crawl Results
When an assessment is in the "Crawled" state, you can manually edit the crawl results by clicking the "Edit Crawl Results" button displayed at the bottom-left of the priority matrix on the assessment page.
On this page, the following operations are available:
- Add endpoints to a detected feature
- Remove endpoints from a detected feature
- Add new features along with their endpoints
Reviewing Results and Running Additional Scans
When the credit threshold is reached during a scan or all selected combinations have been scanned, the assessment pauses in a "Pending" state.
Opening a "Pending" assessment displays the matrix screen again. Each cell shows one of the following states:
| State | Description |
|---|---|
| Scanned | Displayed for combinations where scanning has completed |
| Skipped | Displayed for combinations that were skipped because scanning was deemed unnecessary |
| Priority menu | Displayed for combinations that have not yet been scanned |
From this screen, you can:
- Preview the interim report: Click "Preview Report" to open the current report in a new tab. You can use it to review findings so far and decide whether to continue scanning or complete the assessment
- Run additional scans: Set the credit threshold and priorities for unscanned combinations, then click "Start Pentesting" to run additional scans
- Complete the assessment: If no additional scans are needed, click the "Complete Assessment" button to finalize the assessment
Completing the Assessment
Clicking "Complete Assessment" transitions you to the assessment report page.
Checking Assessment Results
Assessment reports can be viewed on the web, as shown in the following screens.
The format is similar to the output you would receive when requesting an assessment from a security vendor.
Each item in the assessment results explains which feature was assessed from what perspective, and what severity and risk vulnerabilities were found.
How to Start a Re-assessment
From a completed assessment page, click the "Retest" button to choose from the following two modes:
- Retest full security assessment: Re-runs a security assessment against the entire application. A new assessment is created inheriting the original settings (URL, authentication information, etc.), and you can review and modify the settings before starting.
- Retest vulnerable feats or perspectives only: Re-assesses only the features and perspectives where vulnerabilities were detected in the previous assessment. This is suitable for verifying fixes.
In either mode, a configuration screen is displayed before starting the assessment, allowing you to adjust the target and credit threshold in advance.
Assessment Perspectives
Takumi reviews each feature detected through crawling from the following perspectives:
- Injection
- XSS
- CSRF
- SSRF
- File System Vulnerabilities
- Open Redirect
- Broken Authentication
- Broken Authorization
- Business Logic Flaws
- Clickjacking
- CORS Misconfiguration
Each category contains various detailed assessment perspectives. The distinctive feature is that business logic vulnerabilities are particularly included in the assessment scope. Our technology utilizing AI agents enables assessments based on application specifications.
If you want to know more details, meetings with developers are also available. Please feel free to contact us.
Accuracy Evaluation
As a result of performing assessment using only the blackbox assessment feature on a demo application with vulnerabilities embedded independently by GMO Flatt Security, through approximately 20 hours of scanning, the detection rate was 48% and the false positive rate was 33.3%.
Note that the demo application also contains vulnerabilities that are inherently difficult to discover without referring to the source code, and when excluding such vulnerabilities, the detection rate was 70%.