High-Flying Birds

This chapter uses VisiData to answer this question: Which birds that are struck, on average, at the highest altitudes?

In the process, we’ll cover the following skills:

  • Assigning “types” to columns
  • Creating new columns
  • Using frequency tables to filter sheets
  • Calculating aggregate statistics

Note

If you haven’t already, download faa-wildlife-strikes.csv, a dataset of all aircraft-wildlife collisions reported to the Federal Aviation Adminsitration between 2010 and mid-2016.

Open the wildlife-strikes dataset in VisiData

Let’s start from the very beginning.

Run this command in your terminal:

vd faa-wildlife-strikes.csv

If it worked, you should see something like this:

  File  Edit  View  Column  Row  Data  Plot  System  Help     | VisiData 3.0.2 | Alt+H for help men
 OPERATOR            ATYPE         INCIDENT_DATE      STATE  AIRPORT             PHASE_OF_FLT>
 BUSINESS           │ PA-28        │ 05/22/15 00:00:00 │ FL    │ VERO BEACH MUNICIP…│ APPROACH     │
 BUSINESS            BE-1900       06/18/15 00:00:00  AK     KENAI MUNICIPAL AR… APPROACH      BUSINESS            PA-46 MALIBU  09/20/15 00:00:00  TX     DAVID WAYNE HOOKS …               DELTA AIR LINES     B-717-200     11/07/15 00:00:00  MO     LAMBERT-ST LOUIS I… APPROACH      BUSINESS            BE-90 KING    12/17/15 00:00:00  FL     POMPANO BEACH AIRP… LANDING ROLL  DELTA AIR LINES     B-757         07/17/15 00:00:00  VI     HENRY E ROHLSEN AR…               DELTA AIR LINES     B-717-200     08/02/15 00:00:00  TX     SAN ANTONIO INTL    APPROACH      BUSINESS            C-414         08/03/15 00:00:00  TX     LONE STAR EXECUTIV… DEPARTURE     ALLEGIANT AIR       MD-80         09/02/15 00:00:00  FL     TAMPA INTL          APPROACH      TRANS STATES AIRLI… EMB-145       09/07/15 00:00:00  MO     LAMBERT-ST LOUIS I… APPROACH      BUSINESS            C-172         11/28/15 00:00:00  FL     OPA-LOCKA EXECUTIV… APPROACH      GOVERNMENT          EC120         12/08/15 00:00:00  CA     NORMAN Y. MINETA S…               AMERICAN AIRLINES   A-321         05/06/15 00:00:00  FL     FORT LAUDERDALE/HO… APPROACH      EXPRESSJET AIRLINES CRJ100/200    05/06/15 00:00:00  AR     FORT SMITH REGIONA… CLIMB         MESA AIRLINES       CRJ900        05/08/15 00:00:00  AR     BILL AND  HILLARY … LANDING ROLL  BUSINESS            HELICOPTER    05/06/15 00:0┌──────────────────────────────────| statuses |─┐ DELTA AIR LINES     A-320         05/07/15 00:0│ saul.pw/VisiData v3.0.2                        DELTA AIR LINES     A-320         05/08/15 00:0│ opening datasets/faa-wildlife-strikes.csv as  │ LUFTHANSA           A-380         05/10/15 00:0│ csv                                            BUSINESS            C-172         05/08/15 00:0│ Take a breath.                                 SPIRIT AIRLINES     A-319         05/10/15 00:0└───────────────────────────────────────────────┘1› faa-wildlife-strikes|                                                                73448 rows 

Tell VisiData that HEIGHT is an integer column

Navigate to the HEIGHT column, and press #.

You should see something like this:

  File  Edit  View  Column  Row  Data  Plot  System  Help     | VisiData 3.0.2 | Alt+H for help men
<STATE  AIRPORT             PHASE_OF_FLT  HEIGHT# SPEED  SPECIES             BIRDS_STRUCK  >
 FL    │ VERO BEACH MUNICIP…│ APPROACH     │       !│       │ Unknown bird       │ 1            │ N 
 AK     KENAI MUNICIPAL AR… APPROACH            !        Unknown bird        1             N 
 TX     DAVID WAYNE HOOKS …                     !        Unknown bird        1             N 
 MO     LAMBERT-ST LOUIS I… APPROACH            !        Unknown bird        1             N 
 FL     POMPANO BEACH AIRP… LANDING ROLL       0         Unknown bird        1               
 VI     HENRY E ROHLSEN AR…                     !        Unknown bird        1             O 
 TX     SAN ANTONIO INTL    APPROACH            !        Unknown bird        1             N 
 TX     LONE STAR EXECUTIV… DEPARTURE           !        Unknown bird        1             P 
 FL     TAMPA INTL          APPROACH        6000         Unknown bird        1             N 
 MO     LAMBERT-ST LOUIS I… APPROACH            !        Owls                1             N 
 FL     OPA-LOCKA EXECUTIV… APPROACH            !        Hawks               1             N 
 CA     NORMAN Y. MINETA S…                     !        Gulls               1             N 
 FL     FORT LAUDERDALE/HO… APPROACH        1500         Unknown bird - sma… 1             N 
 AR     FORT SMITH REGIONA… CLIMB               !        Unknown bird - sma… 1             N 
 AR     BILL AND  HILLARY … LANDING ROLL       0         Unknown bird - sma… 1             N 
        UNKNOWN             En Route            !        Unknown bird        1             P 
 CA     METRO OAKLAND INTL                      !        Unknown bird        1             N 
 UT     SALT LAKE CITY INTL                     !        Unknown bird        1             N 
 TX     GEORGE BUSH INTERC… CLIMB               !        Unknown bird        1             N 
 FL     ORLANDO SANFORD IN… APPROACH            !        Unknown bird        1             N 
 IL     CHICAGO O'HARE INT… CLIMB          12000         Unknown bird        1             P 
1› faa-wildlife-strikes|                                              #   type-int      73448 rows 

As you can see, many of the reported collisions are missing height data, or appear to have occurred on the ground (i.e., HEIGHT == 0).

So, let’s focus only on collsions reported to have occurred above the ground.

Select only rows where HEIGHT is greater than zero

To do that, we’ll use z|, VisiData’s “select by expression” command. Type z| and then, at the prompt, type HEIGHT > 0. You should see something like this:

  File  Edit  View  Column  Row  Data  Plot  System  Help     | VisiData 3.0.2 | Alt+H for help men
<STATE  AIRPORT             PHASE_OF_FLT  HEIGHT# SPEED  SPECIES             BIRDS_STRUCK  >
 FL    │ VERO BEACH MUNICIP…│ APPROACH     │       !│       │ Unknown bird       │ 1            │ N 
 AK     KENAI MUNICIPAL AR… APPROACH            !        Unknown bird        1             N 
 TX     DAVID WAYNE HOOKS …                     !        Unknown bird        1             N 
 MO     LAMBERT-ST LOUIS I… APPROACH            !        Unknown bird        1             N 
 FL     POMPANO BEACH AIRP… LANDING ROLL       0         Unknown bird        1               
 VI     HENRY E ROHLSEN AR…                     !        Unknown bird        1             O 
 TX     SAN ANTONIO INTL    APPROACH            !        Unknown bird        1             N 
 TX     LONE STAR EXECUTIV… DEPARTURE           !        Unknown bird        1             P 
 FL     TAMPA INTL          APPROACH        6000         Unknown bird        1             N 
 MO     LAMBERT-ST LOUIS I… APPROACH            !        Owls                1             N 
 FL     OPA-LOCKA EXECUTIV… APPROACH            !        Hawks               1             N 
 CA     NORMAN Y. MINETA S…                     !        Gulls               1             N 
 FL     FORT LAUDERDALE/HO… APPROACH        1500         Unknown bird - sma… 1             N 
 AR     FORT SMITH REGIONA… CLIMB               !        Unknown bird - sma… 1             N 
 AR     BILL AND  HILLARY … LANDING ROLL       0         Unknown bird - sma… 1             N 
        UNKNOWN             En Route            !        Unknown bird        1             P 
 CA     METRO OAKLAND INTL                      !        Unknown bird        1             N 
 UT     SALT LAKE CITY INTL                     !        Unknown bird        1             N 
 TX     GEORGE BUSH INTERC… CLIMB               !        Unknown bird        1             N 
 FL     ORLANDO SANFORD IN… APPROACH            !        Unknown bird        1             N 
 IL     CHICAGO O'HARE INT… CLIMB          12000         Unknown bird        1             P 
select by expr: HEIGHT > 0                                        z|   select-expr      73448 rows 

Then press Enter. Now you you should see the above-ground collisions selected:

  File  Edit  View  Column  Row  Data  Plot  System  Help     | VisiData 3.0.2 | Alt+H for help men
<STATE  AIRPORT             PHASE_OF_FLT  HEIGHT# SPEED  SPECIES             BIRDS_STRUCK  >
 FL    │ VERO BEACH MUNICIP…│ APPROACH     │       !│       │ Unknown bird       │ 1            │ N 
 AK     KENAI MUNICIPAL AR… APPROACH            !        Unknown bird        1             N 
 TX     DAVID WAYNE HOOKS …                     !        Unknown bird        1             N 
 MO     LAMBERT-ST LOUIS I… APPROACH            !        Unknown bird        1             N 
 FL     POMPANO BEACH AIRP… LANDING ROLL       0         Unknown bird        1               
 VI     HENRY E ROHLSEN AR…                     !        Unknown bird        1             O 
 TX     SAN ANTONIO INTL    APPROACH            !        Unknown bird        1             N 
 TX     LONE STAR EXECUTIV… DEPARTURE           !        Unknown bird        1             P 
FL    │ TAMPA INTL         │ APPROACH     │   6000 │       │ Unknown bird       │ 1            │ N 
 MO     LAMBERT-ST LOUIS I… APPROACH            !        Owls                1             N 
 FL     OPA-LOCKA EXECUTIV… APPROACH            !        Hawks               1             N 
 CA     NORMAN Y. MINETA S…                     !        Gulls               1             N 
FL    │ FORT LAUDERDALE/HO…│ APPROACH     │   1500 │       │ Unknown bird - sma…│ 1            │ N 
 AR     FORT SMITH REGIONA… CLIMB               !        Unknown bird - sma… 1             N 
 AR     BILL AND  HILLARY … LANDING ROLL       0         Unknown bird - sma… 1             N 
        UNKNOWN             En Route            !        Unknown bird        1             P 
 CA     METRO OAKLAND INTL                      !        Unknown bird        1             N 
 UT     SALT LAKE CITY INTL                     !        Unknown bird        1             N 
 TX     GEORGE BUSH INTERC… CLIMB               !  ┌─────────────────────────────| statuses |─┐ 
 FL     ORLANDO SANFORD IN… APPROACH            !  │ selected 22883 rows                       
IL    │ CHICAGO O'HARE INT…│ CLIMB        │  12000 └──────────────────────────────────────────┘ 
1› faa-wildlife-strikes|                                  z|   select-expr      73448 rows  •22883 

Create a new sheet with only the selected rows

To do so, press ". Once you do that, you should see something like this:

  File  Edit  View  Column  Row  Data  Plot  System  Help     | VisiData 3.0.2 | Alt+H for help men
<STATE  AIRPORT             PHASE_OF_FLT  HEIGHT# SPEED  SPECIES             BIRDS_STRUCK  >
 FL    │ TAMPA INTL         │ APPROACH     │   6000 │       │ Unknown bird       │ 1            │ N 
 FL     FORT LAUDERDALE/HO… APPROACH        1500         Unknown bird - sma… 1             N 
 IL     CHICAGO O'HARE INT… CLIMB          12000         Unknown bird        1             P 
 FL     TAMPA INTL          APPROACH          50         Unknown bird        1             N 
 OR     SOUTHWEST OREGON R… APPROACH         300         Unknown bird        1             N 
 ME     PORTLAND INTL JETP… CLIMB           1000         Unknown bird        1             N 
 VA     ROANOKE REGNL ARPT… APPROACH        3000         Unknown bird        1             N 
 FL     MIAMI INTL          APPROACH         400         Unknown bird        1             N 
 NY     SYRACUSE HANCOCK I… APPROACH         200         Unknown bird        1             N 
        UNKNOWN             EN ROUTE        8000         Unknown bird        1             P 
 PA     PITTSBURGH INTL AR… APPROACH        6000         Unknown bird        1             N 
 MD     BALTIMORE/WASH INT… APPROACH        1000         Unknown bird        1             N 
 NC     RALEIGH-DURHAM INTL APPROACH        1000         Unknown bird        1             N 
 MI     WILLOW RUN ARPT     APPROACH        3000         Unknown bird - lar… 1             N 
 AK     TED STEVENS ANCHOR… APPROACH         100         Unknown bird        1             N 
        UNKNOWN             LOCAL             80         Unknown bird        1             N 
 DC     RONALD REAGAN WASH… APPROACH         100         Unknown bird        1             N 
 AL     BIRMINGHAM-SHUTTLE… APPROACH        1100         Unknown bird        1             N 
 NY     LA GUARDIA ARPT     CLIMB            400         Unknown bird        1             N 
 CA     JOHN WAYNE-ORANGE … APPROACH        1700         Unknown bird - med… 1             N 
 FL     TAMPA INTL          APPROACH        1000         Unknown bird        1             N 
2› faa-wildlife-strikes_selectedref|                                          "         22883 rows 

Get the average collision height for each species

This is a two-step process. First, navigate to the HEIGHT column, and press + to add an aggregator. At the prompt at the bottom of the screen, type mean

  File  Edit  View  Column  Row  Data  Plot  System  Help     | VisiData 3.0.2 | Alt+H for help men
<STATE  AIRPORT             PHASE_OF_FLT  HEIGHT# SPEED  SPECIES             BIRDS_STRUCK  >
 FL    │ TAMPA INTL         │ APPROACH     │   6000 │       │ Unknown bird       │ 1            │ N 
 FL     FORT LAUDERDALE/HO… APPROACH        1500         Unknown bird - sma… 1             N 
 IL     CHICAGO O'HARE INT… CLIMB          12000         Unknown bird        1             P 
 FL     TAMPA INTL          APPROACH          50         Unknown bird        1             N 
 OR     SOUTHWEST OREGON R… APPROACH         300     ┌─────────────────| Choose Aggregators |─┐ 
 ME     PORTLAND INTL JETP… CLIMB           1000     │ Start typing an aggregator name or     │ 
 VA     ROANOKE REGNL ARPT… APPROACH        3000     │ description.                            
 FL     MIAMI INTL          APPROACH         400     │ Multiple aggregators can be added by   │ 
 NY     SYRACUSE HANCOCK I… APPROACH         200     │ separating spaces.                      
        UNKNOWN             EN ROUTE        8000                                              
 PA     PITTSBURGH INTL AR… APPROACH        6000     │ - Enter to select top aggregator.      │ 
                   mean - arithmetic mean of values      │ - Tab to highlight top aggregator.     │ 
                   median - median of values                                                      
                   avg - arithmetic mean of values       ## When Aggregator Highlighted 
                   min - minimum value                                                            
                   max - maximum value                   │ - Tab/Shift+Tab to cycle highlighted   │ 
                   mode - mode of values                 │ aggregator.                             
                   sum - sum of values                   │ - Enter to select aggregators.         │ 
                   distinct - distinct values            │ - Space to add highlighted aggregator. │ 
                   count - number of values              │ - 0-9 to add numbered aggregator.      │ 
                   list - list of values                 └────────────────────────────────────────┘ 
choose aggregators: mean                                         +   aggregate-col      22883 rows 

… and then press Enter.

Note

Adding an aggregator doesn’t change the appearance of the sheet. But if you want to confirm that you’ve added an aggregator correctly, you can press Shift-C to open up the Columns Sheet, and look at the aggregators field.

Then, to get the average for each animal, navigate to the SPECIES column, and press Shift-F to create a frequency sheet. It should look something like this:

  File  Edit  View  Column  Row  Data  Plot  System  Help     | VisiData 3.0.2 | Alt+H for help men
 SPECIES            ↓count♯ percent% histogram                             ~ HEIGHT_mean%     
 Unknown bird       ║  2098 │    9.17 │ ■■■■■■■■■                              │            ! Unknown bird - sma…  8807    38.49  ■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■             ! Unknown bird - lar…   693     3.03  ■■                                                 ! Unknown bird - med…  3056    13.35  ■■■■■■■■■■■■■                                      ! Gulls                 550     2.40  ■■                                                 ! Northern pintail       64     0.28                                                     ! Barred owl              3     0.01                                                     ! Common raven            4     0.02                                                     ! Great blue heron       38     0.17                                                     ! American robin        183     0.80                                                     ! Rock pigeon           219     0.96                                                     ! Eastern meadowlark     77     0.34                                                     ! Hawks                 131     0.57                                                     ! Bald eagle             42     0.18                                                     ! Bonin petrel            3     0.01                                                     ! Killdeer              199     0.87                                                     ! Doves                  72     0.31                                                     ! Turkey vulture        178     0.78                                                     ! Western meadowlark     52     0.23                                                     ! Red-tailed hawk       190     0.83                                                     ! Rough-legged hawk       6     0.03                                                     !3› faa-wildlife-strikes_selectedref_SPECIES_freq|           processing… Shift+F           420 bins 

Sort the species by collision height

By default, frequency tables are sorted by the raw count of matching rows. To sort by the highest average collision height, navigate to the mean_HEIGHT column, and press ].

You should see something like this:

  File  Edit  View  Column  Row  Data  Plot  System  Help     | VisiData 3.0.2 | Alt+H for help men
 SPECIES             count♯↓HEIGHT_mean%                                                        
 Eared grebe        ║     1 │    20000.00  Long-billed curlew      1     15000.00  Whimbrel                1     12000.00  Lesser yellowlegs       4      9925.00  Dusky flycatcher        1      9500.00  Sanderling              1      9500.00  Red-necked phalaro…     4      8525.00  Clark's grebe           1      8500.00  Northern saw-whet …     1      8000.00  Grebes                  1      8000.00  Spotted sandpiper       1      7700.00  Least flycatcher        3      7566.67  Lark sparrow            2      7400.00  Greater yellowlegs      3      7333.33  Pacific-slope flyc…     3      7000.00  Seminole bat            1      7000.00  Solitary sandpiper      2      6250.00  Pocketed free-tail…     1      6000.00  Red-naped sapsucker     2      5750.00  Cinnamon teal           1      5600.00  Horned grebe            2      5400.00 3› faa-wildlife-strikes_selectedref_SPECIES_freq|                    ]   sort-desc        420 bins 

Hrrrm, it seems like a lot of these species show up just a few times — too few to be particularly informative. Let’s do something about that.

Limit the results to relatively common species

This step will seem familiar; it’s a lot like how we selected only above-the-ground collisions.

First, type z| to raise the “select by expression” prompt. Then, type count >= 20:

  File  Edit  View  Column  Row  Data  Plot  System  Help     | VisiData 3.0.2 | Alt+H for help men
 SPECIES             count♯↓HEIGHT_mean%                                                        
 Eared grebe        ║     1 │    20000.00  Long-billed curlew      1     15000.00  Whimbrel                1     12000.00  Lesser yellowlegs       4      9925.00  Dusky flycatcher        1      9500.00  Sanderling              1      9500.00  Red-necked phalaro…     4      8525.00  Clark's grebe           1      8500.00  Northern saw-whet …     1      8000.00  Grebes                  1      8000.00  Spotted sandpiper       1      7700.00  Least flycatcher        3      7566.67  Lark sparrow            2      7400.00  Greater yellowlegs      3      7333.33  Pacific-slope flyc…     3      7000.00  Seminole bat            1      7000.00  Solitary sandpiper      2      6250.00  Pocketed free-tail…     1      6000.00  Red-naped sapsucker     2      5750.00  Cinnamon teal           1      5600.00  Horned grebe            2      5400.00 select by expr: count >= 20                                       z|   select-expr        420 bins 

Next, press Enter to complete the action. Because there are no high-count species in the visible part of the sheet, you won’t notice much of effect at first; just some status messages in the sidebar:

  File  Edit  View  Column  Row  Data  Plot  System  Help     | VisiData 3.0.2 | Alt+H for help men
 SPECIES             count♯↓HEIGHT_mean%            ┌─────────────────────────────| statuses |─┐
 Eared grebe        ║     1 │    20000.00             │ selected 24 rows                          Long-billed curlew      1     15000.00             │ [3x] selected 56 more rows                Whimbrel                1     12000.00             │ [2x] selected 34 more rows                Lesser yellowlegs       4      9925.00             │ [2x] selected 49 more rows                Dusky flycatcher        1      9500.00             │ [2x] selected 39 more rows                Sanderling              1      9500.00             │ selected 693 more rows                    Red-necked phalaro…     4      8525.00             │ selected 23 more rows                     Clark's grebe           1      8500.00             │ selected 183 more rows                    Northern saw-whet …     1      8000.00             │ [2x] selected 32 more rows                Grebes                  1      8000.00             │ selected 3056 more rows                   Spotted sandpiper       1      7700.00             │ selected 2098 more rows                   Least flycatcher        3      7566.67             │ selected 211 more rows                    Lark sparrow            2      7400.00             │ selected 64 more rows                     Greater yellowlegs      3      7333.33             │ selected 53 more rows                     Pacific-slope flyc…     3      7000.00             │ selected 43 more rows                     Seminole bat            1      7000.00             │ [4x] selected 20 more rows                Solitary sandpiper      2      6250.00             │ [3x] selected 25 more rows                Pocketed free-tail…     1      6000.00             │ selected 37 more rows                     Red-naped sapsucker     2      5750.00             │ selected 60 more rows                     Cinnamon teal           1      5600.00             │ [2x] selected 48 more rows                Horned grebe            2      5400.00             └───| Ctrl+P to view all status messages |[┘
3› faa-wildlife-strikes_selectedref_SPECIES_freq|            z|   select-expr        420 bins  •75 

Now press " to create a new sheet containing only the selected rows. Tada!:

  File  Edit  View  Column  Row  Data  Plot  System  Help     | VisiData 3.0.2 | Alt+H for help men
 SPECIES             count♯↓HEIGHT_mean%                                                        
 Yellow-bellied sap…║    24 │     3585.42                                                         
 Swainson's thrush      56      3454.46                                                         
 Snow goose             34      3258.82                                                         
 Hermit thrush          49      2770.10                                                         
 Red-eyed vireo         39      2768.72                                                         
 Unknown bird - lar…   693      2691.92                                                         
 Northern shoveler      23      2602.17                                                         
 American robin        183      2545.00                                                         
 White-throated spa…    56      2426.79                                                         
 Greater white-fron…    32      2335.94                                                         
 Unknown bird - med…  3056      2260.49                                                         
 Unknown bird         2098      2253.93                                                         
 Dark-eyed junco        39      2031.28                                                         
 Perching birds (y)    211      2018.73                                                         
 Northern pintail       64      1980.19                                                         
 American coot          53      1965.66                                                         
 Unknown bird or bat    43      1892.88                                                         
 Red bat                20      1750.00                                                         
 Geese                  56      1696.88                                                         
 Gadwall                25      1675.80                                                         
 Gray catbird           37      1573.27                                                         
4› faa-wildlife-strikes_selectedref_SPECIES_freq_selectedref|                 "            75 bins 

Finally, type g_ to widen all the columns, so that you can read all the names:

  File  Edit  View  Column  Row  Data  Plot  System  Help     | VisiData 3.0.2 | Alt+H for help men
 SPECIES                      count♯↓HEIGHT_mean%                                               
 Yellow-bellied sapsucker    ║    24 │     3585.42                                                
 Swainson's thrush               56      3454.46                                                
 Snow goose                      34      3258.82                                                
 Hermit thrush                   49      2770.10                                                
 Red-eyed vireo                  39      2768.72                                                
 Unknown bird - large           693      2691.92                                                
 Northern shoveler               23      2602.17                                                
 American robin                 183      2545.00                                                
 White-throated sparrow          56      2426.79                                                
 Greater white-fronted goose     32      2335.94                                                
 Unknown bird - medium         3056      2260.49                                                
 Unknown bird                  2098      2253.93                                                
 Dark-eyed junco                 39      2031.28                                                
 Perching birds (y)             211      2018.73                                                
 Northern pintail                64      1980.19                                                
 American coot                   53      1965.66                                                
 Unknown bird or bat             43      1892.88                                                
 Red bat                         20      1750.00                                                
 Geese                           56      1696.88                                                
 Gadwall                         25      1675.80                                                
 Gray catbird                    37      1573.27                                                
4› faa-wildlife-strikes_selectedref_SPECIES_freq_selectedref| g_   resize-cols-max         75 bins