A Python review of NYCHA data raises new questions about the authority's tracking of lead

  • Amongst the documents produced by NYCHA to Progress Queens in response to a FOIL request was a previously unreadable CSV data file that contained over 1 million rows of requests for maintenance repairs.

  • A review of a data file that large was possible by using Python programming language to conduct searches of the data file.

By LOUIS FLORES

New questions are being raised about how the New York City Housing Authority tracks information about the possibility of its tenants being exposed to lead at its public housing developments, according to a review of records provided to Progress Queens by NYCHA.

The data reviewed for this report originated in a CSV file produced by NYCHA to Progress Queens in response to a request made by Progress Queens under the State’s Freedom of Information Law. The CSV file contained rows of information about service requests for repairs with and without work orders.

A CSV file is a raw data file that is formatted to be read by or imported into various computer applications. The CSV file was so large that it was impossible to fully open the file in Excel, a software application for spreadsheets that under normal circumstances can read CSV files. Consequently, the publisher of Progress Queens had to study Python, a programming language, in order to conduct searches of all the data contained in the CSV file. A majority of the information presented in this report was obtained as output generated by a Python module written by the publisher of Progress Queens. To support each of the democraticization of data and an open Government, Progress Queens has publicly published its Python module on GitHub. Using a Python interpreter generated additional analysis to supplement the information provided by the module.

Since NYCHA is reportedly the target of a Federal investigation examining the physical condition standards of its public housing developments, the information on the CSV file is material to understanding how many instances NYCHA has acknowledged lead-related risks at its public housing developments, at least as they relate to service requests for repairs.

After examining the data in the CSV file, Progress Queens compared the output of various searches with information contained in a report published by The New York Daily News about lead paint inspections and abatements.

“NYCHA is required by [the U.S. Department of Housing and Urban Development] to inspect apartments annually, but is not required to proactively test for lead during those inspections unless paint dust or chips are visible during the inspection,” the journalist Greg Smith noted in a report he filed for The New York Daily News.

It is not known how NYCHA treats service requests that are labeled by its own database to be expressly lead-related.

Several requests made by Progress Queens to NYCHA about the FOIL records, including the CSV file reviewed for this report, were not answered.

Mr. Smith’s article for The New York Daily News reportedly included a review of lead paint inspections and abatements performed by NYCHA for the period from 01 January 2013 through 30 March 2015. “During that time, lead paint was detected and abated in 1,604 out of 3,096 inspections at 93 developments across all five boroughs,” Mr. Smith noted in his report.

The data reviewed by Progress Queens for this report included reviewing 1,483,617 rows of data on the CSV file for the period of time beginning on 31 December 2009 and ending on 20 November 2015, according to an analysis of the CSV file using the Python module and other search commands using a Python interpreter.

Of the 1,483,617 service requests on the CSV file, 1,209 service requests were expressly described as being lead-related. A Python search isolated the 1,209 lead-related service requests to 753 unique apartment units, according to identification codes in the CSV file. Because NYCHA has refused to provide cross-reference information to create a look-up table from the identification codes, it was not immediately possible to determine the physical location of the unique apartment units.

Because the distribution of the lead-related service requests were skewed around the year 2010, the lead-related data may be incomplete. It is also noteworthy that the number of annual lead-related service requests did not appear to remain constant, relative to the annual number of total service requests, which did appear to remain constant.

According to the review of lead paint inspections and abatement records performed by The New York Daily News, NYCHA tenants made only 44 requests for lead paint inspection and abatement for the period of time applicable to its review of records.

When Progress Queens applied a search filter to the service requests for descriptions that were expressly labeled as being lead-related, only 25 records were returned for the corresponding period of 01 January 2013 through 30 March 2015, indicating that either NYCHA was not accurately describing service requests that were lead-related in its service request database, that the service request CSV file provided to Progress Queens was incomplete, or that NYCHA was using another system to track lead-related service requests or work orders.

It’s important to note that, in addition to the overall 1,209 lead-related service requests that were expressly labeled to be lead-related on the CSV file, there were an addition 185,807 service requests in the CSV file that were paint-related. It is not known if any of those paint-related service requests may have triggered a lead paint inspection and abatement that NYCHA then failed to retroactively label as being expressly lead-related.

Possible misrepresentation about the risks of lead exposure

The report in The New York Daily News noted that NYCHA has been aware that approximately 55,000 of its 178,000 apartments contain lead-based paint.

On the political front, Mr. Smith, the reporter at The New York Daily News, rightly claimed that the staggering number of apartments known to NYCHA to still contain lead paint contrasted sharply with the testimony provided by NYCHA CEO Shola Olatoye during a March 2016 budget hearing, at which time NYCHA CEO Olatoye said that only 18 apartments had tested positive for lead paint since 2010. NYCHA CEO Olatoye’s testimony was duly noted in a report published by Progress Queens, the incredulity of which prompted the FOIL request that resulted in the production of records that made possible the analysis that is reflected in this report.

In addition to NYCHA CEO Olatoye admitting during City Council testimony that apartments had tested positive for lead paint, still yet further apartments had tested positive for lead in drinking water. However, NYCHA CEO Olatoye failed to identify which apartments tested positive for lead in drinking water, and New York City Councilmember Ritchie Torres (D-Fordham), chair of the City Council committee on public housing, did not ask NYCHA to publicly release that information.

It was not until NYCHA produced some records to Progress Queens in response to the FOIL request that the public finally learned which apartments tested positive for lead in drinking water.

One of the apartments identified in records produced by NYCHA to Progress Queens in response to a FOIL request, where high levels of lead was detected in drinking water, was located at the Metro North Plaza Houses.  Source : Jim Henderson-Wikimedia Commons-Universal (CC0 1.0)

One of the apartments identified in records produced by NYCHA to Progress Queens in response to a FOIL request, where high levels of lead was detected in drinking water, was located at the Metro North Plaza Houses.  Source : Jim Henderson-Wikimedia Commons-Universal (CC0 1.0)

The lack of any demands for political accountability for reported risks of lead-based exposure within NYCHA apartments diverges from the political hot potato that water-bourne contaminants, including lead, have caused in other municipalities, such as in Flint, Michigan, and Hoosick Falls, New York.

The absence of a political firestorm over the risks of lead exposure

As reported by Progress Queens, if NYCHA were to remain in financial limbo and, consequently, were to continue to face large, unfunded capital improvement projects, the municipal housing authority would remain ripe for being privatized by the de Blasio administration. How that is being accomplished by municipal officials is by transferring public housing stock into the Federal Section 8 program for housing subsidies. Once in the Federal Section 8 program, NYCHA apartments can be packaged by City officials for sale to real estate investors, as happened in secret in 2014 to a portfolio of project-based, Section 8 buildings.

AFTER PROGRESS QUEENS EXPOSED THAT A CASH-STRAPPED NYCHA HAD PAID TO MAKE MAJOR REPAIRS TO PROJECT-BASED, SECTION 8 BUILDINGS THAT IT LATER SOLD IN SECRET TO A CONSORTIUM OF DEVELOPERS, INCLUDING A POLITICALLY-CONNECTED REAL ESTATE FIRM, THE OFFICES OF THE NEW YORK CITY COMPTROLLER AND THE DEPARTMENT OF INVESTIGATION WOULD NOT COMMENT ON THE CONDITIONS OF THE SALE.  ABOVE, AT LEFT, IS 930 HALSEY STREET, AND, AT RIGHT, 55 SARATOGA SQUARE, TWO SECTION 8 BUILDINGS FORMERLY OWNED BY NYCHA IN BEDFORD-STUYVESANT, BROOKLYN.  THE BUILDINGS RECEIVED MAJOR IMPROVEMENTS IN THE TIME BEFORE THE BUILDINGS WERE SOLD IN DECEMBER 2014.  THE ABOVE PHOTOGRAPH IS DATED SEPTEMBER 2014, THREE MONTHS BEFORE THE STRUCTURED FINANCE TRANSACTION CLOSED.   SOURCE :  GOOGLE EARTH

AFTER PROGRESS QUEENS EXPOSED THAT A CASH-STRAPPED NYCHA HAD PAID TO MAKE MAJOR REPAIRS TO PROJECT-BASED, SECTION 8 BUILDINGS THAT IT LATER SOLD IN SECRET TO A CONSORTIUM OF DEVELOPERS, INCLUDING A POLITICALLY-CONNECTED REAL ESTATE FIRM, THE OFFICES OF THE NEW YORK CITY COMPTROLLER AND THE DEPARTMENT OF INVESTIGATION WOULD NOT COMMENT ON THE CONDITIONS OF THE SALE.  ABOVE, AT LEFT, IS 930 HALSEY STREET, AND, AT RIGHT, 55 SARATOGA SQUARE, TWO SECTION 8 BUILDINGS FORMERLY OWNED BY NYCHA IN BEDFORD-STUYVESANT, BROOKLYN.  THE BUILDINGS RECEIVED MAJOR IMPROVEMENTS IN THE TIME BEFORE THE BUILDINGS WERE SOLD IN DECEMBER 2014.  THE ABOVE PHOTOGRAPH IS DATED SEPTEMBER 2014, THREE MONTHS BEFORE THE STRUCTURED FINANCE TRANSACTION CLOSED.   SOURCE :  GOOGLE EARTH

At that March budgetary hearing, Councilmember Torres appeared to be more concerned with plans for NYCHA to shift more public housing apartments into the Federal Section 8 program, so that the Section 8 apartments could be privatized, than he was about NYCHA CEO Olatoye revealing which apartments had tested positive for lead in drinking water.

Furthermore, for the report published by The New York Daily News, Councilmember Torres expressed frustration over the pace by which NYCHA was abating lead from its housing developments, but nothing explains why Councilmember Torres failed to raise such concerns during the March budgetary hearing, which he chaired.

Advance questions about the subject matter of this report were submitted by Progress Queens to Councilmember Torres’ staff, but the questions were not answered.

About the FOIL request and NYCHA’s response

In total, NYCHA produced approximately 1 GB of electronic documents to Progress Queens in response to the FOIL request. Separately, NYCHA has reportedly produced over 400 million records of information to the U.S. Attorney’s Office for New York’s southern district, which is reportedly investigating the municipal housing authority over the physical condition standards of its apartments and allegations of possible false financial claims. It is not know how NYCHA arrived at the count of over 400 million records. For example, the CSV file examined for this report contained over 1,4 million rows of data. It is not known if NYCHA considers the CSV file as one file, as the number of pages it would take to print the underlying data, or as 1,4 million rows of information.

After NYCHA over-charged Progress Queens for the production of ten (10) DVD/CD discs of information that could have easily fit onto just one (1) DVD disc, NYCHA summarily closed the FOIL request matter, despite there being several unanswered questions about the adequacy of the records produced, in particular, the glaring discrepancy between the records provided to Progress Queens and the reported size of the records produced to the U.S. Attorney’s Office.

The FOIL request filed by Progress Queens sought all non-exempt records produced by NYCHA to the U.S. Attorney’s Office, such that any records that would be subject to withholding, such as documents about the health records of NYCHA tenants, would not have to be provided by NYCHA to Progress Queens. However, it is clear from the report published by The New York Daily News that records about lead paint inspection and abatement exist, but those records, seemingly responsive to Progress Queens’ FOIL request, were not turned over by NYCHA to Progress Queens.

The U.S. Attorney’s Office has an official policy of not commenting on, much less acknowledging, investigations that Federal prosecutors are reportedly conducting. Notwithstanding that policy, Progress Queens submitted three advance questions to the press office of the U.S. Attorney’s Office with respect to the subject matter of this report. Even though two out of the three questions did not directly pertain to the reported investigation, the press office of the U.S. Attorney’s Office chose not to answer any of the questions.

Recommended Reading

Further Reading

Reference Documents

A note about the CSV file and the Python module

The CSV file examined for this report contained 1,483,617 rows of data, and it included service requests with and without work orders. A separate CSV file, not examined for this report, included 1,442,563 records of only service requests with work orders ; this file was not examined. Both CSV files are publicly available on Google Drive, along with all the other records produced by NYCHA in response to the FOIL request. The main FOIL page on Progress Queens’ Web site contains links to all publicly-available documents obtained by Progress Queens from various public documents requests.

When trying to access the CSV files, Google Drive is unable to preview the CSV file, prompting the option to download the file. Because the CSV file cannot be fully opened in Excel, the publisher of Progress Queens was compelled to study Python in order to run searches of the underlying data. To that end, the publisher of Progress Queens successfully wrote a module that searched the CSV file examined for this report and produced some noteworthy information. To make the publicly-available CSV file meaningful to the public, the publisher of Progress Queens likewise published the module on a public platform. For the module to work, each of the module and the CSV file of service requests with and without work orders must locally exist on a computer, and the module must obviously be modified to reflect the drive of the module and the CSV file. The module was written using Python version 3.5.1 | Anaconda custom (x86_64) | (default, Dec 7 2015, 11:24:55) [GCC 4.2.1 (Apple Inc. build 5577)] and Pandas version 0.18.1 using Anaconda. More information about the module can be found on the read me file available on GitHub.