Truncate noisy build-failure output, fix the recurring terrain death
The web UI was showing a raw ~500k-character stdout/stderr dump on a failed build — go-pmtiles' extract logs one progress line per chunk, which is fine on a terminal (it overwrites itself) but not captured non-interactively. command_output now keeps just the tail, and the status line renders it with preserved line breaks in a scrollable box instead of one unreadable blob. Root cause of the actual failure (terrain build dying ~11 minutes in with an HTTP/2 stream reset): go-pmtiles' default --overfetch=0.05 merged this multi-country planet.pmtiles extract into a single ~60GB request with no way to resume after a dropped stream. --overfetch=0 splits it into many smaller requests instead.
This commit is contained in:
1 parent
237e5b2e54
commit
f18e5c19d3
5 files changed
+92
-13
No files matched your search
@@ -513,7 +513,17 @@ module Maptool
|
||||
extracts = sources.each_with_index.map do |item, i|
|
||||
report_step.call(progress_step("extracting #{item["name"]}", labels, i))
|
||||
extract_path = File.join(scratch_dir, "extract-#{i}.pmtiles")
|
||||
run_pmtiles!("extract", item["url"], extract_path, "--region=#{region_path}", "--maxzoom=#{TERRAIN_MAX_ZOOM}")
|
||||
# --overfetch=0: go-pmtiles' default (0.05) merges nearby byte
|
||||
# ranges into fewer, bigger HTTP requests to save round-trips —
|
||||
# for planet.pmtiles (global z0-12, tens of GB) covering a
|
||||
# multi-country selection that collapsed the whole extract into
|
||||
# a *single* ~60GB request, which then had no way to resume after
|
||||
# an HTTP/2 stream reset 11 minutes in (see the "died again"
|
||||
# 2026-09-12 build). Disabling merging trades that one fragile,
|
||||
# unresumable mega-request for many small ones spread over
|
||||
# --download-threads (default 4) — slower on a clean run, but no
|
||||
# longer one dropped stream away from losing 20 minutes of transfer.
|
||||
run_pmtiles!("extract", item["url"], extract_path, "--region=#{region_path}", "--maxzoom=#{TERRAIN_MAX_ZOOM}", "--overfetch=0")
|
||||
extract_path
|
||||
end
|
||||
|
||||
@@ -621,11 +631,23 @@ module Maptool
|
||||
ENV.fetch("PLANETILER_JAR", "/usr/local/lib/planetiler.jar")
|
||||
end
|
||||
|
||||
# go-pmtiles' extract logs one progress line per chunk fetched
|
||||
# ("fetching N dirs, M chunks, K requests") — fine on a terminal where
|
||||
# it overwrites itself, but captured non-interactively that's one line
|
||||
# per chunk, easily hundreds of thousands of characters for a big
|
||||
# region. The actual failure is always at the end, so keep only the
|
||||
# tail — same story for osmium/planetiler's own chatty stdout.
|
||||
OUTPUT_TAIL_LINES = 40
|
||||
|
||||
# Some CLI tools (go-pmtiles included) write their actual error to
|
||||
# stdout rather than stderr — combine both rather than risk raising a
|
||||
# BuildError with nothing useful in it.
|
||||
def self.command_output(stdout, stderr)
|
||||
[stdout, stderr].map(&:strip).reject(&:empty?).join("\n")
|
||||
lines = [stdout, stderr].map(&:strip).reject(&:empty?).join("\n").lines
|
||||
return lines.join if lines.size <= OUTPUT_TAIL_LINES
|
||||
|
||||
omitted = lines.size - OUTPUT_TAIL_LINES
|
||||
"(#{omitted} earlier lines omitted)\n#{lines.last(OUTPUT_TAIL_LINES).join}"
|
||||
end
|
||||
private_class_method :command_output
|
||||
|
||||
|
||||
Reference in new issue
Block a user