大量目标时如何用spray识别二级反代目录？

[lovelyjuice]

假设我有1000个目标，这些目标均开启了WAF，能够封禁高频目录爆破的IP。我的字典大小为16w，希望一秒钟只对每个目标发送一个请求包，防止IP被封禁。有以下需求：

1. 智能404识别，比如输出结果不应包含自定义的404页面。
2. 重定向识别，有些目标未登录状态访问/abcd/目录，它会给你重定向到/login.jsp?return_url=/abcd/;jsessionid=xxxxx, 这时候输出结果不需要包含/abcd/目录，只保留一个/login.jsp。或者仅保留一个/abcd/目录，同时对于其它的/def/,/xyz/目录，虽然也会重定向到登录页面，但不保留。总而言之只需要保留一个能跳转到登录页面的结果即可。
3. 部分目标访问其它目录和根目录都是404/403，但访问/xtgl/目录，会自动重定向到/xtgl/admin_login.jsp，这时需要保留/xtgl/或者/xtgl/admin_login.jsp。同样是二选一，不能同时保留两者。
4. 输出格式为jsonl，仅保留状态码为200的目录，或者状态码为301/302但满足第2、4点要求的目录，其它401、403、503等不需要。方便后续提取URL使用漏扫进行扫描

spray能否实现以上需求？如果可以的话是否能提供一下命令行参数示例？

[M09Ic]

1. 默认配置下的spray应该已经可以智能识别404
2. 默认配置下的spray应该也可以识别这类重定向跳转，主要随机目录的基准值也是如你所说的跳转， 对于二级目录下的重定向跳转， 需要 spray -u的参数从二级目录开始 例如 spray -u http://example.com/aaa/ , 在下个大版本会提供重构后的智能递归, 可以更智能的选择这类场景递归爆破
3. 默认配置下的spray, 已经支持指定目录的重定向保留
4. 可以指定输出文件, 例如-f 1.jl ,默认格式即为jsonline. 可以使用jq或者脚本过滤
5. spray 支持--rate-limit 这个参数是每个目标独立作用的.

综上 只需要 spray -l 1.txt --rate-limit 10 -d custom.txt -f output.jl 即可.

小建议: spray在大量目标是还是不够稳定, 因此1000个目标建议分成10个一组, 通过脚本分批调用, 遇到的问题欢迎在此issue下追问.

这是个很好的使用场景, spray设计之初就是作为大型扫描工具的目录爆破工件.

[M09Ic]

欢迎在大规模, 分布式, 复杂场景下使用spray, 如果遇到的任何困难, 疑惑, 建议, 需求都可以提出issue, 一同讨论解决实现

[lovelyjuice]

我测试过了，按你的命令行执行，结果文件不是json而是类似与httpx的输出结果

因为还包含终端颜色，所以一堆乱码，需要加-j参数才能json输出
https://chainreactors.github.io/wiki/spray/start/#_12

至于--rate-limit应该要设成 1 才是每个目标每秒只发一个包，我用的参数是--pool 1000 --rate-limit 1 -t 1
https://chainreactors.github.io/wiki/spray/start/#_9

默认情况下404和403无法完全排除，不知道是什么原因，被迫使用了--black-status 400,410,403,404,500,501,502,503

你说的二级目录下的重定向跳转，我正好碰到了，为了避免垃圾数据填充满结果，目前只能--black-status 301,302，后果就是会丢失部分二级目录

还有对于springboot的动态401识别似乎有点问题，我使用-w '{$l#4}' --pool 10 --rate-limit 10 -t 2 -f spray_result_spring.jsonl -j -t 1 --black-status 302,301,400,410,403,404,500,501,502,503进行扫描时，会得到下面的结果

{"number":8,"valid":true,"fuzzy":false,"url":"http://xxx.com:8001/h","path":"/h","host":"","body_length":88,"header_length":374,"status":401,"spend":25,"content_type":"json","title":"json data","frameworks":{"springboot":{"name":"springboot","froms":{"6":true},"tags":["fingers","component"],"is_focus":true,"attributes":{"part":"a","vendor":"","product":"springboot"}}},"extracts":null,"error":"","reason":"","source":7,"depth":0,"distance":5,"unique":18964,"hashes":{"body-md5":"3298bff2d6b0c07af7d4a3633ac700e8","header-md5":"745fcaebb7a3e9bd2df15ec2666cdb56","raw-md5":"d446d38c19cc855ae3adc2d6a6a09857","body-simhash":"bd7d2a1e43cbf6af","header-simhash":"d8f2df0e8623bfed","raw-simhash":"98f2fb0e862bb7ed","body-mmh3":"9710045"}}
{"number":9,"valid":true,"fuzzy":false,"url":"http://xxx.com:8001/i","path":"/i","host":"","body_length":88,"header_length":374,"status":401,"spend":23,"content_type":"json","title":"json data","frameworks":{"springboot":{"name":"springboot","froms":{"6":true},"tags":["fingers","component"],"is_focus":true,"attributes":{"part":"a","vendor":"","product":"springboot"}}},"extracts":null,"error":"","reason":"","source":7,"depth":0,"distance":5,"unique":18964,"hashes":{"body-md5":"95c6be818a20b94a11dfe8d52c34f600","header-md5":"0d49a2dce7b299d8788e9f193e9eca63","raw-md5":"82bd925a06d809068893285d56da9185","body-simhash":"bd7d2b1e47cbf6a6","header-simhash":"d8f2df0e8623bfed","raw-simhash":"98f2db0e8623b7ed","body-mmh3":"-186533766"}}
{"number":16,"valid":true,"fuzzy":false,"url":"http://xxx.com:8001/p","path":"/p","host":"","body_length":88,"header_length":374,"status":401,"spend":3,"content_type":"json","title":"json data","frameworks":{"springboot":{"name":"springboot","froms":{"6":true},"tags":["fingers","component"],"is_focus":true,"attributes":{"part":"a","vendor":"","product":"springboot"}}},"extracts":null,"error":"","reason":"","source":7,"depth":0,"distance":5,"unique":18964,"hashes":{"body-md5":"9f8fa9b051b823f030bf972802b4ba3f","header-md5":"0e65d02302958786e61d22d353b4a4d4","raw-md5":"9e8b427ed4e2626048f951b7573af4d8","body-simhash":"bd75ea1e43ebf6ab","header-simhash":"d8f2df0e8623b7ed","raw-simhash":"98f2fb0e8623b7e9","body-mmh3":"-696624264"}}
{"number":47,"valid":true,"fuzzy":false,"url":"http://xxx.com:8001/au","path":"/au","host":"","body_length":89,"header_length":374,"status":401,"spend":15,"content_type":"json","title":"json data","frameworks":{"springboot":{"name":"springboot","froms":{"6":true},"tags":["fingers","component"],"is_focus":true,"attributes":{"part":"a","vendor":"","product":"springboot"}}},"extracts":null,"error":"","reason":"","source":7,"depth":0,"distance":5,"unique":18964,"hashes":{"body-md5":"e6fa6185c962909d12d2c7e7ab2040da","header-md5":"3b66773f820e34f98e817fb64565edba","raw-md5":"74435b02d25b48e1cce332e67e5720e1","body-simhash":"bd746a1e77eb76af","header-simhash":"d8f2dd0e8623b7ed","raw-simhash":"98f2e91e862bb7ed","body-mmh3":"1862565826"}}
{"number":49,"valid":true,"fuzzy":false,"url":"http://xxx.com:8001/aw","path":"/aw","host":"","body_length":89,"header_length":374,"status":401,"spend":133,"content_type":"json","title":"json data","frameworks":{"springboot":{"name":"springboot","froms":{"6":true},"tags":["fingers","component"],"is_focus":true,"attributes":{"part":"a","vendor":"","product":"springboot"}}},"extracts":null,"error":"","reason":"","source":7,"depth":0,"distance":5,"unique":18964,"hashes":{"body-md5":"165721c59fe0f9889549576378b9595e","header-md5":"bcd4d9adff97273368e6de61cf2b3afc","raw-md5":"c6446994c336b48acf1f8510dc397458","body-simhash":"bd756a1e77ebf6af","header-simhash":"d8f2dd0e8623b7ed","raw-simhash":"98f2e91e862bb7ed","body-mmh3":"-445361964"}}
{"number":62,"valid":true,"fuzzy":false,"url":"http://xxx.com:8001/bj","path":"/bj","host":"","body_length":89,"header_length":374,"status":401,"spend":3,"content_type":"json","title":"json data","frameworks":{"springboot":{"name":"springboot","froms":{"6":true},"tags":["fingers","component"],"is_focus":true,"attributes":{"part":"a","vendor":"","product":"springboot"}}},"extracts":null,"error":"","reason":"","source":7,"depth":0,"distance":5,"unique":18964,"hashes":{"body-md5":"0cf995562f1b95b4e1c801cacb858d62","header-md5":"807b91c0cbacccf9c47161df53649589","raw-md5":"1732cedc624787a331880b49a9a22c47","body-simhash":"b9756a1e53ebf62f","header-simhash":"d8f2dd0e8623b7ed","raw-simhash":"98f2e91e862bb7ed","body-mmh3":"-1544295490"}}
{"number":103,"valid":true,"fuzzy":false,"url":"http://xxx.com:8001/cy","path":"/cy","host":"","body_length":89,"header_length":374,"status":401,"spend":3,"content_type":"json","title":"json data","frameworks":{"springboot":{"name":"springboot","froms":{"6":true},"tags":["fingers","component"],"is_focus":true,"attributes":{"part":"a","vendor":"","product":"springboot"}}},"extracts":null,"error":"","reason":"","source":7,"depth":0,"distance":5,"unique":18964,"hashes":{"body-md5":"c16b61bd0a10fcc28eb8db0cacb9ae99","header-md5":"10677977d71f86375b330491ecca1c20","raw-md5":"e69a07a45c83cd3f312473ac4d1fd5f8","body-simhash":"b97c681e43eb762f","header-simhash":"d8f2dd0e8623b7ed","raw-simhash":"98f2e91e862bb7ed","body-mmh3":"-610911067"}}
{"number":162,"valid":true,"fuzzy":false,"url":"http://xxx.com:8001/ff","path":"/ff","host":"","body_length":89,"header_length":374,"status":401,"spend":2,"content_type":"json","title":"json data","frameworks":{"springboot":{"name":"springboot","froms":{"6":true},"tags":["fingers","component"],"is_focus":true,"attributes":{"part":"a","vendor":"","product":"springboot"}}},"extracts":null,"error":"","reason":"","source":7,"depth":0,"distance":6,"unique":18964,"hashes":{"body-md5":"7bd56e367e879e1cf3b251d485f30295","header-md5":"ccf6e83b87e043ce94ea878372e0c1e0","raw-md5":"f3e16121cf0e32f6aacc314be699a3d7","body-simhash":"387d6a1e67eb66af","header-simhash":"d8f2dd0e8623b7ed","raw-simhash":"98f2e90e862bb7ed","body-mmh3":"514380115"}}
{"number":166,"valid":true,"fuzzy":false,"url":"http://xxx.com:8001/fj","path":"/fj","host":"","body_length":89,"header_length":374,"status":401,"spend":3,"content_type":"json","title":"json data","frameworks":{"springboot":{"name":"springboot","froms":{"6":true},"tags":["fingers","component"],"is_focus":true,"attributes":{"part":"a","vendor":"","product":"springboot"}}},"extracts":null,"error":"","reason":"","source":7,"depth":0,"distance":5,"unique":18964,"hashes":{"body-md5":"2ed4e4f00890feba815878d8b1a8d1b3","header-md5":"ccf6e83b87e043ce94ea878372e0c1e0","raw-md5":"868caa85e965e1bbbaa19e502ddc2d67","body-simhash":"387d6a1e67ebf623","header-simhash":"d8f2dd0e8623b7ed","raw-simhash":"98f2e90e862bb7e9","body-mmh3":"737147937"}}
{"number":168,"valid":true,"fuzzy":false,"url":"http://xxx.com:8001/fl","path":"/fl","host":"","body_length":89,"header_length":374,"status":401,"spend":3,"content_type":"json","title":"json data","frameworks":{"springboot":{"name":"springboot","froms":{"6":true},"tags":["fingers","component"],"is_focus":true,"attributes":{"part":"a","vendor":"","product":"springboot"}}},"extracts":null,"error":"","reason":"","source":7,"depth":0,"distance":5,"unique":18964,"hashes":{"body-md5":"685987c3be0ab9830bb11f698550281a","header-md5":"d32d24196c016547c38ca636bed953b9","raw-md5":"79d652a3317e40a71a2d8a11cbbe675a","body-simhash":"387d6a1e67ebeeab","header-simhash":"d8f2dd0e8623b7ed","raw-simhash":"98f2e90e862bb7e9","body-mmh3":"1240075889"}}
{"number":176,"valid":true,"fuzzy":false,"url":"http://xxx.com:8001/ft","path":"/ft","host":"","body_length":89,"header_length":374,"status":401,"spend":3,"content_type":"json","title":"json data","frameworks":{"springboot":{"name":"springboot","froms":{"6":true},"tags":["fingers","component"],"is_focus":true,"attributes":{"part":"a","vendor":"","product":"springboot"}}},"extracts":null,"error":"","reason":"","source":7,"depth":0,"distance":5,"unique":18964,"hashes":{"body-md5":"f6d26876056b6df7544a42f84009f6fd","header-md5":"d32d24196c016547c38ca636bed953b9","raw-md5":"f39549b31875bd5e437ebc76f7961776","body-simhash":"397d6a1e63eb76af","header-simhash":"d8f2dd0e8623b7ed","raw-simhash":"98f2c90e862bb7ed","body-mmh3":"2110481929"}}
{"number":187,"valid":true,"fuzzy":false,"url":"http://xxx.com:8001/ge","path":"/ge","host":"","body_length":89,"header_length":374,"status":401,"spend":60,"content_type":"json","title":"json data","frameworks":{"springboot":{"name":"springboot","froms":{"6":true},"tags":["fingers","component"],"is_focus":true,"attributes":{"part":"a","vendor":"","product":"springboot"}}},"extracts":null,"error":"","reason":"","source":7,"depth":0,"distance":5,"unique":18964,"hashes":{"body-md5":"d43aa6c18d8b629703e14d7d4e5e958a","header-md5":"f66280a1daf3d266a77e9a1b5630f1b0","raw-md5":"b5a32c114d3dd71eef9062ffc8e101eb","body-simhash":"3874681e67eb6eaf","header-simhash":"d8f2dd0e8623b7ed","raw-simhash":"98f2f90e862bb7ed","body-mmh3":"1006295276"}}
{"number":188,"valid":true,"fuzzy":false,"url":"http://xxx.com:8001/gf","path":"/gf","host":"","body_length":89,"header_length":374,"status":401,"spend":36,"content_type":"json","title":"json data","frameworks":{"springboot":{"name":"springboot","froms":{"6":true},"tags":["fingers","component"],"is_focus":true,"attributes":{"part":"a","vendor":"","product":"springboot"}}},"extracts":null,"error":"","reason":"","source":7,"depth":0,"distance":6,"unique":18964,"hashes":{"body-md5":"f8c136e7d373fc1196bd4ae381f97875","header-md5":"f66280a1daf3d266a77e9a1b5630f1b0","raw-md5":"83b546b341b89155692c79953c34a9e3","body-simhash":"39756a1e73eb7eae","header-simhash":"d8f2dd0e8623b7ed","raw-simhash":"98f2e90e862bb7ed","body-mmh3":"566821279"}}
{"number":189,"valid":true,"fuzzy":false,"url":"http://xxx.com:8001/gg","path":"/gg","host":"","body_length":89,"header_length":374,"status":401,"spend":7,"content_type":"json","title":"json data","frameworks":{"springboot":{"name":"springboot","froms":{"6":true},"tags":["fingers","component"],"is_focus":true,"attributes":{"part":"a","vendor":"","product":"springboot"}}},"extracts":null,"error":"","reason":"","source":7,"depth":0,"distance":5,"unique":18964,"hashes":{"body-md5":"95d170e0f566fea145c6695b8e94425a","header-md5":"f66280a1daf3d266a77e9a1b5630f1b0","raw-md5":"1cd46efcdc5835228344e8bc0a0fe34d","body-simhash":"397d691e73eb66ab","header-simhash":"d8f2dd0e8623b7ed","raw-simhash":"98f2e90e862bb7e9","body-mmh3":"910659578"}}
{"number":190,"valid":true,"fuzzy":false,"url":"http://xxx.com:8001/gh","path":"/gh","host":"","body_length":89,"header_length":374,"status":401,"spend":2,"content_type":"json","title":"json data","frameworks":{"springboot":{"name":"springboot","froms":{"6":true},"tags":["fingers","component"],"is_focus":true,"attributes":{"part":"a","vendor":"","product":"springboot"}}},"extracts":null,"error":"","reason":"","source":7,"depth":0,"distance":5,"unique":18964,"hashes":{"body-md5":"0288c77d3d98da62dad3ef40cc838bb7","header-md5":"f66280a1daf3d266a77e9a1b5630f1b0","raw-md5":"54165b8467164c9807e9538718805dcd","body-simhash":"39756a1e73eb7ea2","header-simhash":"d8f2dd0e8623b7ed","raw-simhash":"98f2e90e862bb7e9","body-mmh3":"115385505"}}
{"number":195,"valid":true,"fuzzy":false,"url":"http://xxx.com:8001/gm","path":"/gm","host":"","body_length":89,"header_length":374,"status":401,"spend":3,"content_type":"json","title":"json data","frameworks":{"springboot":{"name":"springboot","froms":{"6":true},"tags":["fingers","component"],"is_focus":true,"attributes":{"part":"a","vendor":"","product":"springboot"}}},"extracts":null,"error":"","reason":"","source":7,"depth":0,"distance":5,"unique":18964,"hashes":{"body-md5":"2a9a2211069d7cfd7cfe928dbc96abb4","header-md5":"f66280a1daf3d266a77e9a1b5630f1b0","raw-md5":"c14b73bdc2cfadfafabfbdb3fcbc1db4","body-simhash":"3974691e73ebeea3","header-simhash":"d8f2dd0e8623b7ed","raw-simhash":"98f2e90e862bb7e9","body-mmh3":"486784556"}}
{"number":196,"valid":true,"fuzzy":false,"url":"http://xxx.com:8001/gn","path":"/gn","host":"","body_length":89,"header_length":374,"status":401,"spend":3,"content_type":"json","title":"json data","frameworks":{"springboot":{"name":"springboot","froms":{"6":true},"tags":["fingers","component"],"is_focus":true,"attributes":{"part":"a","vendor":"","product":"springboot"}}},"extracts":null,"error":"","reason":"","source":7,"depth":0,"distance":5,"unique":18964,"hashes":{"body-md5":"e49650263070a2aecad1e5b5347f9d40","header-md5":"712090ae565c6c8523c0d1b89cafd7b9","raw-md5":"5bbbf944c6ee2a1446cfdac889e486e1","body-simhash":"39756a1e73eb7ea2","header-simhash":"d8f2dd0e8623b7e9","raw-simhash":"98f2e90e862bb7e9","body-mmh3":"-69913799"}}
{"number":197,"valid":true,"fuzzy":false,"url":"http://xxx.com:8001/go","path":"/go","host":"","body_length":89,"header_length":374,"status":401,"spend":3,"content_type":"json","title":"json data","frameworks":{"springboot":{"name":"springboot","froms":{"6":true},"tags":["fingers","component"],"is_focus":true,"attributes":{"part":"a","vendor":"","product":"springboot"}}},"extracts":null,"error":"","reason":"","source":7,"depth":0,"distance":5,"unique":18964,"hashes":{"body-md5":"23308ee3ed314b38584d5bbeba04c644","header-md5":"712090ae565c6c8523c0d1b89cafd7b9","raw-md5":"91158a156db86783aef8d469d9efa6ab","body-simhash":"3974691e73eb66ab","header-simhash":"d8f2dd0e8623b7e9","raw-simhash":"98f2e90e862bb7e9","body-mmh3":"223230133"}}
{"number":207,"valid":true,"fuzzy":false,"url":"http://xxx.com:8001/gy","path":"/gy","host":"","body_length":89,"header_length":374,"status":401,"spend":3,"content_type":"json","title":"json data","frameworks":{"springboot":{"name":"springboot","froms":{"6":true},"tags":["fingers","component"],"is_focus":true,"attributes":{"part":"a","vendor":"","product":"springboot"}}},"extracts":null,"error":"","reason":"","source":7,"depth":0,"distance":5,"unique":18964,"hashes":{"body-md5":"d176b673a269964e4f84a34981af83df","header-md5":"1fe1d934ffacc9b2da80a4bbc8f94476","raw-md5":"0934d70fce25023d9e2edfabe2f177f6","body-simhash":"3d7d6b1e57eb7ea3","header-simhash":"d8f2dd0e8623b7e9","raw-simhash":"98f2e90e862bb7e9","body-mmh3":"1553550388"}}
{"number":208,"valid":true,"fuzzy":false,"url":"http://xxx.com:8001/gz","path":"/gz","host":"","body_length":89,"header_length":374,"status":401,"spend":3,"content_type":"json","title":"json data","frameworks":{"springboot":{"name":"springboot","froms":{"6":true},"tags":["fingers","component"],"is_focus":true,"attributes":{"part":"a","vendor":"","product":"springboot"}}},"extracts":null,"error":"","reason":"","source":7,"depth":0,"distance":5,"unique":18964,"hashes":{"body-md5":"3f05f6abe02c36fc8fe72f57d8071f8a","header-md5":"1fe1d934ffacc9b2da80a4bbc8f94476","raw-md5":"8ed2e82fecc17fe02b97a375222a5db6","body-simhash":"3d756b1e57eb7eb3","header-simhash":"d8f2dd0e8623b7e9","raw-simhash":"98f2e90e862bb7e9","body-mmh3":"-1323775040"}}
{"number":209,"valid":true,"fuzzy":false,"url":"http://xxx.com:8001/ha","path":"/ha","host":"","body_length":89,"header_length":374,"status":401,"spend":1,"content_type":"json","title":"json data","frameworks":{"springboot":{"name":"springboot","froms":{"6":true},"tags":["fingers","component"],"is_focus":true,"attributes":{"part":"a","vendor":"","product":"springboot"}}},"extracts":null,"error":"","reason":"","source":7,"depth":0,"distance":5,"unique":18964,"hashes":{"body-md5":"c7f5f229784131714e51d5b5a3b52756","header-md5":"1fe1d934ffacc9b2da80a4bbc8f94476","raw-md5":"4227005442bad878572af25f031d7b0d","body-simhash":"3d756a1e57ebf2ab","header-simhash":"d8f2dd0e8623b7e9","raw-simhash":"98f2e90e862bb7e9","body-mmh3":"836722117"}}
{"number":210,"valid":true,"fuzzy":false,"url":"http://xxx.com:8001/hb","path":"/hb","host":"","body_length":89,"header_length":374,"status":401,"spend":3,"content_type":"json","title":"json data","frameworks":{"springboot":{"name":"springboot","froms":{"6":true},"tags":["fingers","component"],"is_focus":true,"attributes":{"part":"a","vendor":"","product":"springboot"}}},"extracts":null,"error":"","reason":"","source":7,"depth":0,"distance":5,"unique":18964,"hashes":{"body-md5":"3d07440ac3735c719dca28ecbec2b3d1","header-md5":"1fe1d934ffacc9b2da80a4bbc8f94476","raw-md5":"316c7722f40cbed5f84c4f65b38fcbfe","body-simhash":"3d756a1e57ebe6a7","header-simhash":"d8f2dd0e8623b7e9","raw-simhash":"98f2e90e862bb7e9","body-mmh3":"-941566524"}}
{"number":211,"valid":true,"fuzzy":false,"url":"http://xxx.com:8001/hc","path":"/hc","host":"","body_length":89,"header_length":374,"status":401,"spend":3,"content_type":"json","title":"json data","frameworks":{"springboot":{"name":"springboot","froms":{"6":true},"tags":["fingers","component"],"is_focus":true,"attributes":{"part":"a","vendor":"","product":"springboot"}}},"extracts":null,"error":"","reason":"","source":7,"depth":0,"distance":5,"unique":18964,"hashes":{"body-md5":"ccbfdca417c7dfa09cade589c67ea839","header-md5":"1fe1d934ffacc9b2da80a4bbc8f94476","raw-md5":"9162f068ca2bdbed1615fe734fd209c3","body-simhash":"3d756a1e57eb7abe","header-simhash":"d8f2dd0e8623b7e9","raw-simhash":"98f2e90e862bb7e9","body-mmh3":"-1444835716"}}
{"number":212,"valid":true,"fuzzy":false,"url":"http://xxx.com:8001/hd","path":"/hd","host":"","body_length":89,"header_length":374,"status":401,"spend":2,"content_type":"json","title":"json data","frameworks":{"springboot":{"name":"springboot","froms":{"6":true},"tags":["fingers","component"],"is_focus":true,"attributes":{"part":"a","vendor":"","product":"springboot"}}},"extracts":null,"error":"","reason":"","source":7,"depth":0,"distance":5,"unique":18964,"hashes":{"body-md5":"9df23c800bafa4f69f4edc8b835164d9","header-md5":"1fe1d934ffacc9b2da80a4bbc8f94476","raw-md5":"efdb2dbfba8baf5db97f1cef58bedae0","body-simhash":"3d756a1e57eb6ebb","header-simhash":"d8f2dd0e8623b7e9","raw-simhash":"98f2e90e862bb7e9","body-mmh3":"-359465649"}}
{"number":213,"valid":true,"fuzzy":false,"url":"http://xxx.com:8001/he","path":"/he","host":"","body_length":89,"header_length":374,"status":401,"spend":3,"content_type":"json","title":"json data","frameworks":{"springboot":{"name":"springboot","froms":{"6":true},"tags":["fingers","component"],"is_focus":true,"attributes":{"part":"a","vendor":"","product":"springboot"}}},"extracts":null,"error":"","reason":"","source":7,"depth":0,"distance":5,"unique":18964,"hashes":{"body-md5":"23cc2a5dd86f771fb2f55cc73f0966b9","header-md5":"1fe1d934ffacc9b2da80a4bbc8f94476","raw-md5":"b446d481d9e3bfdb5885061c13c4131d","body-simhash":"3d7c6a1e57eb6aab","header-simhash":"d8f2dd0e8623b7e9","raw-simhash":"98f2e90e862bb7e9","body-mmh3":"-826594728"}}
{"number":214,"valid":true,"fuzzy":false,"url":"http://xxx.com:8001/hf","path":"/hf","host":"","body_length":89,"header_length":374,"status":401,"spend":3,"content_type":"json","title":"json data","frameworks":{"springboot":{"name":"springboot","froms":{"6":true},"tags":["fingers","component"],"is_focus":true,"attributes":{"part":"a","vendor":"","product":"springboot"}}},"extracts":null,"error":"","reason":"","source":7,"depth":0,"distance":5,"unique":18964,"hashes":{"body-md5":"915d08f79752a2d8e2eafe51c2656aef","header-md5":"1fe1d934ffacc9b2da80a4bbc8f94476","raw-md5":"859550a6b78d4379d1e553c200353b74","body-simhash":"3d7c6a1e57eb76b3","header-simhash":"d8f2dd0e8623b7e9","raw-simhash":"98f2e90e862bb7e9","body-mmh3":"1224783252"}}
{"number":215,"valid":true,"fuzzy":false,"url":"http://xxx.com:8001/hg","path":"/hg","host":"","body_length":89,"header_length":374,"status":401,"spend":3,"content_type":"json","title":"json data","frameworks":{"springboot":{"name":"springboot","froms":{"6":true},"tags":["fingers","component"],"is_focus":true,"attributes":{"part":"a","vendor":"","product":"springboot"}}},"extracts":null,"error":"","reason":"","source":7,"depth":0,"distance":5,"unique":18964,"hashes":{"body-md5":"8e22fa408afdec96cbe72239a78a6158","header-md5":"c265692125cfa391f2e2d25be5fb81ec","raw-md5":"06d21bd33e658cdcf340ee39e7a5cd22","body-simhash":"3d74681e57eb76a2","header-simhash":"d8f2db0e8623b7ed","raw-simhash":"98f2e90e862bb7e9","body-mmh3":"1275136583"}}
{"number":225,"valid":true,"fuzzy":false,"url":"http://xxx.com:8001/hq","path":"/hq","host":"","body_length":89,"header_length":374,"status":401,"spend":36,"content_type":"json","title":"json data","frameworks":{"springboot":{"name":"springboot","froms":{"6":true},"tags":["fingers","component"],"is_focus":true,"attributes":{"part":"a","vendor":"","product":"springboot"}}},"extracts":null,"error":"","reason":"","source":7,"depth":0,"distance":5,"unique":18964,"hashes":{"body-md5":"5f40096f26ad98b42e9e6ee50ee7fcdd","header-md5":"202f15c398927b1c2d3b77888a5af053","raw-md5":"715ef2d76aedf2dfa353ffa451a0be1f","body-simhash":"3c7d681e53ebeaae","header-simhash":"d8f2db0e8623b7ed","raw-simhash":"98f2c90e862bb7ed","body-mmh3":"286502901"}}
{"number":227,"valid":true,"fuzzy":false,"url":"http://xxx.com:8001/hs","path":"/hs","host":"","body_length":89,"header_length":374,"status":401,"spend":102,"content_type":"json","title":"json data","frameworks":{"springboot":{"name":"springboot","froms":{"6":true},"tags":["fingers","component"],"is_focus":true,"attributes":{"part":"a","vendor":"","product":"springboot"}}},"extracts":null,"error":"","reason":"","source":7,"depth":0,"distance":5,"unique":18964,"hashes":{"body-md5":"62a964697b8ff841a37f400e9f54c242","header-md5":"202f15c398927b1c2d3b77888a5af053","raw-md5":"39b30d5e4b81b7bd5d94318c3705f9a5","body-simhash":"3d75681e63ebeea2","header-simhash":"d8f2db0e8623b7ed","raw-simhash":"98f2e90e862bb7e9","body-mmh3":"716335742"}}
{"number":233,"valid":true,"fuzzy":false,"url":"http://xxx.com:8001/hy","path":"/hy","host":"","body_length":89,"header_length":374,"status":401,"spend":6,"content_type":"json","title":"json data","frameworks":{"springboot":{"name":"springboot","froms":{"6":true},"tags":["fingers","component"],"is_focus":true,"attributes":{"part":"a","vendor":"","product":"springboot"}}},"extracts":null,"error":"","reason":"","source":7,"depth":0,"distance":5,"unique":18964,"hashes":{"body-md5":"b355144d180f0c6d8cd9684c80b5f998","header-md5":"202f15c398927b1c2d3b77888a5af053","raw-md5":"7b197293ff42072329735d0e81012074","body-simhash":"3d7d6a1e63eb76af","header-simhash":"d8f2db0e8623b7ed","raw-simhash":"98f2eb0e862bb7ed","body-mmh3":"1138304347"}}

[M09Ic]

1. 输出文件中不是json是近期版本重构输出时的bug, 临时解决办法可以按照你说的解决
2. 没有排除的404和403是否可以确认下 fuzzy是否为true, 如果不为true, 是否可以提供一个案例？
3. 我重新检查了关于重定向的智能判断， 确实存在一定问题，现在已经讲301/302设置为默认的fuzzyStatus. 具体逻辑为: 如果发现301/302状态码， 会尝试与fuzzy基准值（第一次发现301/302的包）中的进行对比判断， 如果重定向url不为空, 且与base不相同, 则判断为有效目录， 否则自动过滤。 之前因为没将301/302加入fuzzyStatus， 所以这条规则没生效。
4. 401目录与(3)同理， 如果random基准值是401， 则可以自动过滤，否则需要手动设置401为fuzzyStatus. 我看你的结果中判定为有效目录， 可能是random不为401.

上面提到的改动会在下个版本中发布。

[M09Ic]

关于1和3的修复, 已在 https://github.com/chainreactors/spray/releases/tag/v1.1.0 中release, 该release为prerelease, 可能会出现bug.

[lovelyjuice]

第4点 springboot 401状态码判定异常的url：
68747470733a2f2f6e65746375742e636e2f31693636613276756f，临时剪贴板有效期3天

[M09Ic]

裁剪版中似乎没有内容.

有个通用的解决办法是状态码不加入到--black-status 而是加入到--fuzzy-status. 这样在数据能保留到文件中, 在命令行中不输出(valid为false, 但fuzzy为true), 用作后续脚本手动分析.

也有一个隐藏的快捷用法 --fuzzy-status all 打开所有状态码的fuzzy判定.

[lovelyjuice]

现在应该有内容了

Details

[M09Ic]

新的id是多少呢？ 68747470733a2f2f6e65746375742e636e2f31693636613276756f 中并没有内容

[lovelyjuice]

新的id是多少呢？ 68747470733a2f2f6e65746375742e636e2f31693636613276756f 中并没有内容

https://netcut.cn/1i66a2vuo 这个剪贴板里面看不到内容吗？

[M09Ic]

好像还是不行， 可以通过邮件给我发送相关信息或联系方式， m09ician@gmail.com

[M09Ic]

找到问题了， springboot错误页面中存在时间这个随机值。 在总体文本量较小的情况下， simhash计算出来的结果为5-8. 默认的阈值是5， 所以默认的过滤策略失效了。

可以通过sim-distance 9 提高这个阈值. 后续考虑将默认阈值设置为8或者7