Graybox Assessment

info

The English user guide is currently in beta preview. Most of the documents have been automatically translated from the Japanese version. Should you find any inaccuracies, please reach out to Flatt Security.

Overview

Takumi's graybox assessment feature receives both the source code of an application and the URL of the running instance that serves it. Takumi analyzes the source code to find candidate vulnerabilities, attacks the running application to confirm which of them actually reproduce, and outputs the results as a report on the web.

It can be used from the Shisho Cloud byGMO web interface.

Only candidate vulnerabilities that dynamic verification reproduced are reported as findings. The rest are recorded in a separate section of the report, so you can read the vulnerabilities that reproduced apart from those that did not. See Checking Assessment Results for details.

Organization or Target Ownership Verification

Graybox assessments send requests to a running application, so before starting one you need to verify your organization or prove ownership of the target application. For details, please refer to "Pre-Assessment Organization Verification or Ownership Verification".

How to Start an Assessment

Click "Assessments" in the sidebar, then click the "Create Assessment" button in the upper right corner of the screen to start an assessment.

Carry Over Settings from an Existing Assessment

When you want to run a new assessment with the same settings as a previous one, you can carry over the configuration of an existing assessment instead of filling in the form from scratch. Follow these steps:

1. Click the "Create Assessment" button in the upper right corner of the assessment list.
2. Select "Create from existing assessment" in the dialog.
3. Choose the assessment whose settings you want to carry over.

The creation form opens pre-filled with the selected assessment's configuration. Review and adjust the settings, then start the assessment.

You can also use this feature to retry an assessment that failed during execution, keeping the same settings.

Assessment Configuration

The main items on the configuration screen are:

• Assessment Name: Enter a name to identify this assessment.
• Report Language: Select the language of the assessment report (English or Japanese).
• Assessment Method: Select "Graybox Assessment".

Assessment Type

You can select one of the following two modes:

• Full Assessment: Feature enumeration and the initial scan of the target source code are performed in one seamless process. After completion, the assessment pauses in a "Pending" state.
• Scoped Assessment: Features are first enumerated from the target codebase. Once complete, the enumerated features are displayed.

Credit Threshold

Credit thresholds can be set separately for the feature enumeration and scanning phases (for "Scoped Assessment" mode, only feature enumeration). Takumi performs feature enumeration and scanning within the specified credit thresholds.

The minimum for the scan threshold is 40 credits, which is higher than for a whitebox assessment. A graybox scan splits this budget between static analysis and the dynamic verification that follows it, so it needs more room to reach a verified result.

Source Code Configuration

You can provide source code using one of the following methods:

GitHub Repositories

You can specify one or more GitHub repositories. Click the "Add Repositories" button to add repositories for assessment. You can also specify the branch of the target repository at this point. If not specified, the repository's default branch will be used for assessment.

File Upload

You can upload source code archives directly. This is useful when GitHub integration is not available or for assessing local codebases.

• Supported formats: .zip or .tar.gz
• Click the "Upload Files" button to select and upload your archive
• The extracted root directory will be displayed and used for file scope specification below

File Scopes

You can specify file paths to include in or exclude from the assessment. All file paths support glob patterns (e.g., src/auth/**, backend/services/**). Leaving the scope empty will assess the entire codebase.

Include Scopes

• Target: Select the target repository or uploaded file.
• Feature Type (optional): Specify a feature category (e.g., authentication, payments, api).
• File Path: Specify file paths to include in the assessment.

Please note that even if specified as an assessment target, file paths may be excluded based on Takumi's judgment.

Exclusions

• Target: Select the target repository or uploaded file.
• Reason (optional): Document the reason for exclusion (e.g., "Test code", "Auto-generated files").
• File Path: Specify file paths to exclude.

Application Settings

These settings describe the running application that dynamic verification works against.

• Target URL: Enter the URL of the running application that serves the source code you selected above. You can add more than one, which is useful when a single source tree serves several origins (for example, a web front end and its API on separate hosts).
• Out-of-Scope URLs: Specify URLs to exclude from the scope of dynamic verification.
• In-App Authentication: Configure the credentials used to sign into the application. You can provide an ID and password, or select "Other" and describe the authentication procedure in free form. "Account Type" (for example, admin or regular user) tells Takumi which role the credentials represent.

info

Out-of-Scope URLs are used to decide what dynamic verification treats as a target. Even a URL that is out of scope may still be accessed by Takumi during the assessment. Note that this setting does not completely block such access.

Reviewing Feature Enumeration Results

When feature enumeration completes in "Scoped Assessment" mode, a matrix of detected features and assessment perspectives is displayed. On this screen, you can set the credit threshold for scanning and assessment priorities for each feature and perspective.

You can set priorities of "Auto, High, Medium, Low, None" for each combination of feature and perspective. Higher-priority items are scanned first. When "Auto" is selected, Takumi automatically determines the priority based on risk analysis.

After completing the configuration, click the "Start Pentesting" button to begin the scan.

Reviewing Results and Running Additional Scans

When the specified credit threshold is reached during a scan or scanning for all combinations is complete, the assessment pauses in a "Pending" state.

Opening a "Pending" assessment displays the matrix screen again. Each cell shows one of the following states:

State	Description
Scanned	Displayed for combinations where scanning has completed
Skipped	Displayed for combinations that were skipped because scanning was deemed unnecessary
Priority menu	Displayed for combinations that have not yet been scanned

From this screen, you can:

• Preview the interim report: Click "Preview Report" to open the current report in a new tab
• Run additional scans: Set the credit threshold and priorities for unscanned combinations, then click "Start Pentesting" to run additional scans
• Complete the assessment: If no additional scans are needed, click the "Complete Assessment" button to finalize the assessment

Checking Assessment Results

Assessment reports can be viewed on the web. Each item in the assessment results explains which feature was assessed from what perspective, what severity and risk the vulnerability carries, and how dynamic verification confirmed it.

Findings are split across the following sections:

• Findings List contains the vulnerabilities that dynamic verification reproduced. Each finding carries a verification result describing what the verification established, followed by the reproduction steps it performed. The steps are written so that you can follow them yourself to see the vulnerability.
• Unconfirmed Findings contains the candidate vulnerabilities that static analysis raised but dynamic verification could not reproduce. Reproduction steps are not included here, because nothing was reproduced.

Exporting a PDF Report

You can download the assessment report as a PDF with a branded cover page. Click the "Issue PDF Report" button on the assessment report page and choose a cover page language (English or Japanese). Once the PDF is ready, a download link will be sent to the email address associated with your account.

Download links expire after 15 minutes. If a link expires, click the download button again to get a new one. Note that issued PDFs are deleted after 30 days, so save a local copy if needed.

How to Start a Re-assessment

From a completed assessment page, click the "Retest" button to choose from the following two modes:

• Retest the entire application: Runs the assessment again from feature enumeration. The assessment creation screen opens pre-filled with the original settings (repositories, target URLs, authentication information, etc.), and you can review and modify them before starting. The new assessment is created only when you start it.
• Retest the entire application (reuse crawl results): Assesses the entire application again, carrying over the original settings and feature enumeration results so the enumeration is skipped. A new assessment is created in the enumerated state, and you can review and adjust the features and perspectives to assess before starting.

In either mode, the source code is fetched from GitHub again when the assessment runs, so the latest code of the configured branch (or the default branch if none is set) is assessed. If the features of the application have changed significantly since the original assessment, choose "Retest the entire application".

Re-assessment is available only for assessments whose source code is specified as GitHub repositories. Uploaded files are kept only for a limited time, so assessments created from uploaded files cannot be retested.

Impact on the Assessment Target

A graybox assessment verifies the vulnerability candidates raised by static analysis by attacking the running application. Takumi does not test for service outages and caps the rate at which it sends requests, but depending on the target's configuration and state, unexpected load or behavior may still occur — availability is not guaranteed to be unaffected.

During an assessment, accounts and data may also be created on the target application, and existing data may be updated or deleted.

For these reasons, we recommend assessing a staging environment isolated from production wherever possible.

Handling of Assessment Data and Authentication Credentials

Assessment data — the assessment settings (including the configured authentication credentials) and execution results such as feature enumeration results, execution messages, and reports — can be viewed only by users who belong to the organization that created the assessment and hold one of the following roles.

• Owner (organization/owner)
• Takumi manager (organization/takumi_manager)
• Takumi user (organization/takumi_user)

Creating and running an assessment is likewise available to users holding one of these roles.

Of these roles, only users with the "Owner" or "Takumi manager" role can delete an assessment.

The configured credentials are retained as part of the assessment settings so that Takumi can log into the target application during dynamic verification, and so that the settings can be carried over to a new assessment created from an existing one. The retained credentials can be viewed on the assessment settings screen, and may also appear unmasked in messages that describe the progress of an assessment.

For logging into the target application, we recommend using accounts prepared specifically for the assessment.

Deleting an Assessment

To delete an assessment, click "Assessments" in the sidebar to open the assessment list, click the menu icon on the right side of the target assessment's row, and select "Delete". In the confirmation dialog, type delete and click the "Delete" button.

Note that deleting an assessment also deletes its associated data, including the assessment settings (with the configured credentials), feature enumeration results, and reports. Deletion cannot be undone — once deleted, an assessment and its data cannot be restored.

Assessment Perspectives

Static analysis in a graybox assessment reviews the code from the same perspectives as a whitebox assessment.

Feature-Level Assessment

• Injection
• File System Vulnerabilities
• XSS
• Broken Authorization
• Business Logic Flaws

Repository-Level Assessment

• Misconfiguration
• Broken Authentication

Vulnerabilities such as Injection and XSS are tested for each feature identified by Takumi in advance to ensure the comprehensiveness of the assessment. On the other hand, perspectives such as Misconfiguration and Broken Authentication, which tend to exist in specific locations (e.g., configuration files or middleware layers) rather than being scattered across individual features, are executed only once for the entire target.

Each candidate vulnerability raised from these perspectives is then attacked against the running application, and only those that reproduce are reported as findings. Whether one reproduces depends on the state of the running application, so a candidate that static analysis raises correctly may still fail to reproduce — for example, when the affected feature requires data or a privilege the provided credentials do not have.

For the correspondence between these perspectives and OWASP ASVS 5.0, see Compliance with OWASP ASVS 5.0.

Regarding Credit Consumption

Credits are required to use this feature. Credit consumption varies depending on the characteristics and size of the assessment target.

You can set credit thresholds for the feature enumeration phase and the scanning phase separately when starting and when resuming an assessment. Each phase's consumption will not exceed its specified threshold.

As a rough guide, each combination of a "feature" and a "perspective" (corresponding to a specific cell on the configuration screen) typically consumes approximately 5–8 credits. However, actual consumption also varies depending on factors such as the scale of the target feature and the number of clues Takumi discovers during the assessment, so it may fall outside this range.

Note that while an assessment is running, the credit limit you specified is reserved from your balance in advance, so the displayed credit balance is the value after deducting the reserved amount. See Credit Reservation While a Run Is in Progress for details.

info

Even if actual credit consumption exceeds the configured threshold, the excess credits will not be charged.

For example, if the credit threshold is set to 10 and the actual consumption is 11, the final billed credit consumption will be 10.