Debugging VisiData¶
Not everything goes according to plan. To oversimplify somewhat, you may encounter three sources of errors in VisiData:
- Expected errors, the result of doing something you know will fail in certain situations.
- User-generated errors, the result of issuing commands that are invalid in some way.
- Bugs, which stem from problems in VisiData’s own source code.
These errors can occur in two different scopes:
- Cell errors pertain to specific, individual cells. These are often (but not always) expected errors.
- General errors relate not to the data itself, but to broader usage patterns. These are often (but not always) user-generated errors.
This chapter describes how to investigate errors at both scopes. A quick guide:
| Keystroke(s) | Action |
|---|---|
| Control-e | Examine the latest general error |
| g + Control-e | Examine all general errors encountered |
| z + Control-e | Examine the cell-specific error under your cursor |
Examining cell errors¶
In the Understanding Columns chapter, we’ve already seen an example of what a cell-specific error looks like, when we told VisiData that the HEIGHT column values of the wildlife-strikes dataset should be integers (using #):
File Edit View Column Row Data Plot System Help Ctrl+H for help menu <STATE | AIRPORT | PHASE_OF_FLT | HEIGHT#| SPEED | SPECIES | BIRDS_STRUCK | > FL | VERO BEACH MUNICIP…| APPROACH | !| | Unknown bird | 1 | … AK | KENAI MUNICIPAL AR…| APPROACH | !| | Unknown bird | 1 | … TX | DAVID WAYNE HOOKS …| | !| | Unknown bird | 1 | … MO | LAMBERT-ST LOUIS I…| APPROACH | !| | Unknown bird | 1 | … FL | POMPANO BEACH AIRP…| LANDING ROLL | 0 | | Unknown bird | 1 | VI | HENRY E ROHLSEN AR…| | !| | Unknown bird | 1 | … TX | SAN ANTONIO INTL | APPROACH | !| | Unknown bird | 1 | … TX | LONE STAR EXECUTIV…| DEPARTURE | !| | Unknown bird | 1 | … FL | TAMPA INTL | APPROACH | 6000 | | Unknown bird | 1 | … MO | LAMBERT-ST LOUIS I…| APPROACH | !| | Owls | 1 | … FL | OPA-LOCKA EXECUTIV…| APPROACH | !| | Hawks | 1 | … CA | NORMAN Y. MINETA S…| | !| | Gulls | 1 | … FL | FORT LAUDERDALE/HO…| APPROACH | 1500 | | Unknown bird - sma…| 1 | … AR | FORT SMITH REGIONA…| CLIMB | !| | Unknown bird - sma…| 1 | … AR | BILL AND HILLARY …| LANDING ROLL | 0 | | Unknown bird - sma…| 1 | … | UNKNOWN | En Route | !| | Unknown bird | 1 | … CA | METRO OAKLAND INTL | | !| | Unknown bird | 1 | … UT | SALT LAKE CITY INTL| | !| | Unknown bird | 1 | … TX | GEORGE BUSH INTERC…| CLIMB | !| | Unknown bird | 1 | … FL | ORLANDO SANFORD IN…| APPROACH | !| | Unknown bird | 1 | … IL | CHICAGO O'HARE INT…| CLIMB | 12000 | | Unknown bird | 1 | … 1› faa-wildlife-strikes| # type-int 73448 rows •0
VisiData obeyed but marked some cells with a ! error/warning annotation — specifically the empty cells, because Python considers it impossible to convert nothingness a whole number.
In some cases, we can intuit why a cell is marked with a !. In other cases, we might want to know more. To do so, navigate to the cell and type z + Control-e. If we do this on the second cell of the HEIGHT column, here’s what we’ll see:
File Edit View Column Row Data Plot System Help Ctrl+H for help menu text ║ Traceback (most recent call last): ║ File "lib/python3.8/site-packages/visidata/wrappers.py", line 108, in wrapply ║ return func(*args, **kwargs) ║ ValueError: invalid literal for int() with base 10: '' ║ zCtrl+E› faa-wildlife-strikes_cell_error| zCtrl+E 4 lines •0
If you use Python frequently, this output may look familiar; it’s the “stack traceback” for the error that VisiData encountered while trying to convert the blank cell (an empty string) to an integer. If you don’t use Python frequently or at all, you may still be able to get a sense of the issue at hand; the final line is usually the most descriptive.
The error sheet is like any other sheet in VisiData, so you can quit it and return to the previous sheet by pressing q.
Note
There are, in fact, two types of cell errors:
- Errors caused by type conversions, as demonstrated above. VisiData flags these with a yellow exclamation mark.
- Errors stemming from dynamic column expressions, such as
=HEIGHT / 0. VisiData flags these with a red exclamation mark.
Examining general errors¶
Let’s try creating a general error on purpose. Say we want to find all OPERATOR values with parentheses in them. We might navigate over to that column, press | to select rows by a pattern, and type ( at the prompt. If we did so, we’d be greeted by an error message at the bottom of the screen:
BUSINESS | C-172 | 05/08/15 00:00:00 | FL | ORLANDO SANFORD IN…| APPROACH ║ SPIRIT AIRLINES | A-319 | 05/10/15 00:00:00 | IL | CHICAGO O'HARE INT…| CLIMB ║ 1› faa-wildlife-strikes| re.error: missing ), unterminated | select-col-regex 73448 rows •0
Why’s that? In Python, these patterns (“regular expressions”) treat parentheses as special characters that group other characters, rather than literal parentheses. For every opening parenthesis, valid patterns require a closing parenthesis to match it. Hence, the error message we see at the bottom of the screen, suggesting that we have an “unterminated” pattern that lacks a closing parenthesis. (Python’s built-in module for processing regular expressions is named re, hence the re.error in the message.)
To examine this general error in greater detail, we can press Control-e — now, or at any point until you trigger another error — to reveal its full traceback in a VisiData text sheet:
File Edit View Column Row Data Plot System Help Ctrl+H for help menu text File "lib/python3.8/site-packages/visidata/threads.py", line 206, in _toplevelTryFunc t.status = func(*args, **kwargs) File "lib/python3.8/site-packages/visidata/selection.py", line 50, in select for r in (Progress(rows, 'selecting') if progress else rows): File "lib/python3.8/site-packages/visidata/selection.py", line 73, in <genexpr> self.select((self.rows[i] for i in rowIdxs), progress=False) File "lib/python3.8/site-packages/visidata/search.py", line 26, in searchRegex vd.searchContext["regex"] = re.compile(regex, sheet.regex_flags()) or vd.error('invalid regex… File ".pyenv/versions/3.8.1/lib/python3.8/re.py", line 250, in compile return _compile(pattern, flags) File ".pyenv/versions/3.8.1/lib/python3.8/re.py", line 302, in _compile p = sre_compile.compile(pattern, flags) File ".pyenv/versions/3.8.1/lib/python3.8/sre_compile.py", line 764, in compile p = sre_parse.parse(p, flags) File ".pyenv/versions/3.8.1/lib/python3.8/sre_parse.py", line 948, in parse p = _parse_sub(source, state, flags & SRE_FLAG_VERBOSE, 0) File ".pyenv/versions/3.8.1/lib/python3.8/sre_parse.py", line 443, in _parse_sub itemsappend(_parse(source, state, verbose, nested + 1, File ".pyenv/versions/3.8.1/lib/python3.8/sre_parse.py", line 836, in _parse raise source.error("missing ), unterminated subpattern", re.error: missing ), unterminated subpattern at position 0 Ctrl+E› errors_recent| Shift+G go-bottom 22 lines •0
Tip
You can type g + Control-e to open a sheet listing all general errors you have encountered during your current VisiData session.
How to report a bug¶
If you encounter an error that seems to come from a bug within VisiData itself (rather than an invalid command), follow VisiData’s “Help and Support” instructions. The developers ask that bug reports include a “command log” that reproduces the issue, so that they can investigate it with more precision. The next chapter demonstrates how to do so.