Frequently Asked Questions
The English user guide is currently in beta preview. Most of the documents have been automatically translated from the Japanese version. Should you find any inaccuracies, please reach out to Flatt Security.
This page answers frequently asked questions about Takumi Pentesting.
Running Assessments and Credits
Does running an assessment in steps with a small credit threshold reduce its accuracy or coverage?
Setting a small credit threshold and resuming the assessment from Pending again and again gives the same accuracy and coverage as running it at once. The credit threshold only controls how far Takumi proceeds with each combination of feature and perspective.
However, Takumi runs a preparation step each time the assessment resumes, so it may consume slightly more credits than running it at once.
Can Takumi detect vulnerabilities that span multiple features?
An assessment basically works within each combination of feature and perspective, but it investigates the impact of a finding and checks its validity across multiple features, so missed detections are not expected to become more likely. If you want to investigate whether combining vulnerabilities across multiple features leads to a compromise, you can use the penetration test feature after the assessment is completed. For details, see Penetration Test.
Can I stop a running assessment, or change its credit threshold?
To stop a running assessment, open the assessment list and select Cancel from the menu of its row. Users with the Owner or Takumi manager role can cancel an assessment.
Note that you cannot change the credit threshold of a running assessment.
How many credits and how much time will an assessment take to complete?
You can check rough figures on the matrix screen before starting the assessment. The matrix screen appears when you open an assessment that has finished its crawl or feature enumeration in Scoped Assessment, or an assessment in Pending.
The matrix screen shows the following two estimates.
- Estimated credit usage: The range of credits needed to assess the selected combinations, and all unassessed combinations.
- Estimated time until the scan stops: The range of time it takes to assess the selected combinations within the credit threshold you set, until the run stops.
Both values are estimates calculated by multiplying the per-combination figures by the number of combinations. The actual credits and duration may exceed these ranges. For the credits per combination, see Credit Consumption Estimates.
Can I see how many credits each assessment consumed?
Yes. On the report page shown after an assessment is completed, the Metadata section in the sidebar shows the total credits the assessment consumed.
Are credits consumed when an assessment or crawl fails?
No. Credits are charged only when an assessment or crawl makes progress and completes successfully.
Check the credits actually consumed under Usage history in Settings > Plan & Billing in the Shisho Cloud console. For a completed assessment, its report page also shows the credits it consumed. For details, see Can I see how many credits each assessment consumed?
Why did my credit balance drop as soon as I started an assessment, and come back 24 hours later?
The credits that dropped were not consumed but reserved for the assessment. When an assessment starts, credits equal to its credit threshold are reserved from the balance. This reservation prevents assessments from running at the same time on a scale beyond the balance. While the credits are reserved, the displayed balance excludes them.
The reservation is released when the assessment ends. However, for convenience, the reservation is released automatically 24 hours after the assessment starts, even while it is still running, and the reserved credits return to the available balance.
In either case, only the credits the assessment actually used are charged. For details, see Credit Reservation While a Run Is in Progress.
Assessment Scope
Does Takumi always stay within the scope in assessments and penetration tests?
Takumi is controlled to stay within the scope, but the possibility of a deviation cannot be eliminated entirely.
We pass the policy for the specified scope to Takumi and have it refer to the policy throughout the whole process. Takumi is required to act only on resources explicitly specified as targets, and to treat everything else as out of scope in principle. Within our own evaluations, Takumi has basically not deviated from the scope.
A deviation can occur mainly when content in the target steers Takumi's decisions (so-called prompt injection), or when an in-scope URL redirects Takumi out of scope automatically. Neither case is likely to have a serious impact.
In penetration tests, a separate system, including an AI model, evaluates Takumi's actions during the test and detects out-of-scope access. Even if a deviation occurs, we can learn its extent and impact early. For how to define the scope, see Scope.
Why are only some features assessed in a source code assessment?
In whitebox and graybox assessments, Crawl Result Filtering is enabled by default. While it is enabled, Takumi filters the features to prioritize during feature enumeration based on the setting, so some features are excluded from the crawl results.
To assess all features, disable the filtering or raise the limit before starting the assessment.
Why are some combinations skipped even though I set a priority?
Takumi decides, for each combination of feature and perspective, whether it needs to be tested. A combination it judges unnecessary is not assessed, and the matrix screen shows it as Skipped.
Which perspectives are assessed on which features is defined for each assessment method. For example, a blackbox assessment checks broken authentication only on features that provide a means of authentication, such as sign-in and sign-up forms. Whitebox and graybox assessments check misconfiguration and broken authentication only once, against the whole repository.
For details on which combinations are skipped, see "Assessment Perspectives" on the page of each assessment method.
Can I exclude some features from a blackbox assessment?
No setting excludes specific features automatically. Choose Scoped Assessment as the assessment type, and once the crawl finishes, edit the crawl results to remove those features or endpoints.
Impact on the Target
Can I prevent operations that could stop the service in a penetration test?
The Destructive Actions setting of a penetration test lets you choose how much Takumi is allowed to perform operations that cause a large-scale outage or irreversible data loss. With Avoid, Takumi avoids destructive actions. Even when such an action is the only remaining path, Takumi ends the test without achieving the objective.
However, even with Avoid, we cannot guarantee that no outage occurs, because the target system may behave unexpectedly depending on its state or configuration.
Signing In
Can Takumi run assessments and penetration tests with accounts that use SSO or multi-factor authentication?
Yes. Takumi signs in by operating a browser as a user does, so it basically supports any authentication that completes in a browser. When you set a credential, choose Existing account under Account setup, enter the ID and password, and add the information needed in Additional Info.
- Accounts that use SSO: Enter the ID and password of the account used for SSO, such as a Google account. Describing the SSO steps in Additional Info makes the sign-in more likely to succeed.
- Accounts that use TOTP-based multi-factor authentication: Enter the TOTP secret (a string in the
otpauth://format) in Additional Info. Takumi generates the one-time password when it signs in.
For a penetration test that assumes a developer account has been taken over, you can also choose Other under Account setup. Enter the access token or other credentials assumed to have leaked in Additional Info.
What should I do if the authentication has a special specification or flow?
Describe the specification and steps of the authentication in detail in Additional Info of the credential. Takumi interprets the description and signs in by following those steps. For example, you can describe the following.
- Basic authentication required before entering the ID and password
- The URL of the sign-in page and the steps to sign in
- How to obtain a token, and the headers to add to requests
If the sign-in fails in an assessment, Takumi interrupts the crawl or the assessment and shows the reason on the assessment page. Review Additional Info and the credential based on that reason, and then run the assessment again. A failed sign-in consumes no credits.
If the target restricts access by IP address or uses a WAF, allow access from Takumi. For the source IP addresses, see IP Addresses Used by Takumi.
What should I do if the sign-in page has a CAPTCHA?
Takumi has no feature to bypass CAPTCHAs. A CAPTCHA is a protection provided by a third party, and bypassing it could violate the terms of service or raise other ethical issues. For the same reason, there is no recommended way to bypass one.
To start the assessment from a signed-in state, you can pass a session cookie instead. Choose Other under Account setup of the credential, and enter the session cookie in Additional Info. Takumi sets the cookie in the browser, which may let the assessment start from the signed-in state.
Note, however, that if the session expires during the assessment, Takumi can no longer stay signed in. In that case, the assessment may be unable to continue.
Reports and Notifications
Can I check the progress of a running assessment or penetration test?
Yes. How you check it differs between assessments and penetration tests.
For an assessment, open the page of the running assessment. Assessment progress shows how many of the combinations planned for the run have been assessed. The count is updated about every 5 minutes, so reload the page to see the latest progress.
For a penetration test, click View progress on the page of the test to open the progress screen. The progress screen shows a timeline of the attack milestones in chronological order, and an attack map that visualizes the attack paths. For details, see Monitoring Progress.
Can I see the vulnerabilities detected so far while an assessment is running?
No, you cannot see the vulnerabilities detected so far while an assessment is running.
As an assessment proceeds, Takumi finds candidate vulnerabilities and verifies whether each one is actually a vulnerability. Some candidates remain unverified in the middle of a run, so which vulnerabilities to report is not finalized until the run ends.
To review results along the way, start the assessment with a smaller credit threshold. When the threshold is reached, the assessment stops in Pending. In this state, Preview Report shows a report of the results for what has been assessed so far. After reviewing it, specify a credit threshold and resume the assessment. Running an assessment in steps like this does not change its accuracy or coverage.
Can I download the report of a pending assessment?
Yes, as Markdown. Download it from the report you open with Preview Report. However, this report contains interim results, not final ones.
A PDF report can be issued only after the assessment is completed.
Can I choose a private Slack channel as a notification destination?
Yes. Invite the Takumi app to the private channel. After the invitation, it may take up to 8 hours for the channel to appear among the destinations.
Data Handling
When is uploaded source code deleted?
It is deleted 30 days after the last assessment run with that file. For a completed assessment, you can also delete it immediately with Delete Source Code under Data management on the report page. For details, see Whitebox Assessment.
When you specify a GitHub repository, Takumi fetches the source code from the repository for each assessment, and discards it when the assessment ends.
Which region are uploaded source code and other data stored in?
Customer data is stored in Japan regions by default. However, some operational metadata and SaaS-side logs follow the hosting region of each SaaS provider. For details, see Vendor Security.
Note that inference for the AI features may use regions of the inference providers outside Japan. The reason is explained in Provisions for AI/LLM Usage.
Is assessment data used to train AI models?
No. We use only inference providers that have confirmed, in their terms of service and agreements such as a DPA, that input data is not used to train AI models.
We ask inference providers not to retain data wherever possible. However, some models with strong cybersecurity capabilities require data to be retained for a period to prevent misuse. Even then, the retained data is not used for training. For details, see Vendor Security.