A change in government often brings significant shifts in policy. Major initiatives taken up by a previous administration can be slowed or reversed, and information that was once publicly available may be taken down or censored. The White House webpage provides some clear examples of this phenomenon. Following the inauguration this past January, press reported that the Trump Administration White House homepage underwent some changes, such as striking references to climate change and removing a spanish language option. Fortunately, if a user wants to view the content from the White House homepage of President Barack Obama, it is still possible to do so by navigating to https://obamawhitehouse.archives.gov.
Citizens can also use the Internet Archive “Wayback Machine” to access www.WhiteHouse.gov and see content for any given day going back several years. These archive solutions are helpful for viewing web content, but hosted files on these pages still have the potential to get lost. Documents that are hosted on pages can become inaccessible as other content is changed.
Since 2010 the Publish What You Pay coalition, academics, industry, investors and other actors submitted hundreds of comment letters to the Securities and Exchange Commission (SEC) to influence the agency’s Section 1504 rulemaking. Every single comment that has been submitted to the SEC is available on the regulatory agency’s website. The comments are available as pdf files on four separate comment records: 2010, 2010-2012, 2013-2015, and 2015-2016. Because of the current wave of government self-censorship, we wanted to make sure we could preserve the evidence in the Section 1504 record. This post will provide the steps to download all linked documents, such as pdf files, from a website. The SEC comment record will be used as an example, but the same steps can be used to download and preserve files hosted on any site.
As with other data scraping and organizing processes, the steps described in this post could be carried out manually. For example, scraping data from a company pdf report can be done manually, with a user entering in data line by line into a spreadsheet, but that is a time-consuming process. As we described previously on Extract-A-Fact, there are tools to help speed up data scraping. To automate the downloading of all linked files on a website, we will use the Google Chrome extension, Chrono Download Manager - see the tutorial below.
Step 1 - Install the Chrome extension
Navigate to the Chrome web store page for the Chrono Download Manager and click the ‘Add to Chrome’ button in the upper right. A notice will pop up and you can safely click ‘Add extension’ to confirm installation. When the installation completes you should find a new icon in the upper right corner of your Chrome browser.
Step 2 - Download linked files
Before proceeding, we recommend you set a dedicated folder for downloads. Navigate to chrome://settings in your Chrome browser and set a specific downloads folder. See the image below for an example.
Next, navigate to the page with the files you intend to download. In this case we will use the most recent 1504 comment record. Once on the page, click the Chrono Download Manager icon in the upper right. Select the ‘Document’ tab in the window that pops up.
The ‘Document’ window presents a list of all the links on the page that are interpreted as documents. In this case, we are only concerned with downloading the pdf files. To narrow the selection, click the ‘pdf’ check box as shown below.
Once you’ve selected all the relevant documents you can click ‘Start all’ in the lower right of the window to download the files into the folder you selected in the Chrome browser settings.
*Optional Step 3 - Categorize the downloaded files
If you follow the steps above you will be able to successfully download all of the files from a webpage, which will simply be listed by their filename (e.g. s72515-1.pdf). To help organize the files, you can have Chrono Download Manager automatically attach the descriptive text corresponding to each file. Click the first document highlighted in green (see image above), scroll down to the last pdf and press shift+left mouse button on the last highlighted pdf. With all of the pdf files checkmarked and selected, click the the ‘Task Properties’ tab as shown below.
Click the text box next to ‘Naming Mask’ and select ‘*text*.*ext*’ then click ‘Start All’ to download all of the files. You’ll find that the downloaded files will now appear in the folder with a descriptive title (e.g. Jana L. Morgan, Director, Publish What You Pay – United States) rather than the numbered file name.
Glencore’s $75 Million Payments Show Why the EITI Must Enforce Project-level Reporting by Oil and Mining Companies
By Dominic Eagleton, Global Witness
This post originally appeared on GlobalWitness.org on March 7, 2017
Tomorrow the Extractive Industries Transparency Initiative (EITI), a global scheme of 51 implementing countries that aims to stop corruption in the oil, gas and mining industries, will meet in Bogota to make critical decisions about a game-changing measure known as project-level reporting.
The EITI requires oil, gas and mining companies to report details about the trillions of dollars they pay to governments across the world for the rights to natural resources. The reporting requirement applies equally to governments who must publish the receipts, enabling citizens in resource-rich countries to follow the money and ensure it’s used for their benefit instead of lining the pockets of corrupt elites.
In most cases however, EITI reporting doesn’t go into enough detail. Payments are generally disclosed only at the company level. So if a company operates five oil projects in a country, often its payments from each project will be lumped together and reported as an aggregate sum. Although this kind of reporting is a step in the right direction, it still allows companies or governments to hide payments from individual projects, which can reach into billions of dollars from a single project – and can easily go missing.
A new Global Witness investigation highlights the need for EITI countries to ensure payments are reported separately for each oil, gas or mining project. Our research shows that between 2013 and 2016 the mining giant Glencore redirected over $75 million to Dan Gertler, a controversial businessman accused of bribing senior officials in Democratic Republic of Congo to advance his mining interests.
The payments arose from Glencore’s Katanga mining project in Congo and, under the terms of the original contract, should have been transferred to the Congolese state. Instead the money was redirected to Dan Gertler’s Africa Horizons company, registered in the Cayman Islands.
Glencore’s disclosures in Congo’s EITI reports – a relatively rare example of project-level reporting in the EITI – played a critical role in revealing that the payments were being redirected. Glencore admitted the payments were made to Gertler. Gertler contests accusations of wrongdoing in his dealings in Congo.
This case highlights why all 51 countries that follow the EITI’s rules need to implement project-level reporting. Corruption happens at the project level in the oil and mining industries, and there’s no way of know for sure where the money is going unless payments are reported at this level of detail.
Pressure from campaign groups including Global Witness prompted the EITI to adopt project-level reporting in the EITI’s global standard in 2013. But the measure has been strongly resisted by a small number of oil companies including Chevron and Exxon, who advocate a much weaker standard for disclosing payments. After three years’ of stalling, project-level reporting has still not been implemented across EITI countries.
There’s no reason why this can’t be done. A small number of EITI countries have published reports that include project-level payment disclosures, including Indonesia, Ghana and Iraq, as well as Congo. Since 2013, 30 countries have introduced laws that compel their oil, gas and mining companies to report payments at the project level, including the UK, Canada, France and Norway. In the UK alone, over 80 extractive companies have disclosed project-level payments worth around $100 billion since 2015, with no ill effects.
The EITI is lagging behind. Moving at this embarrassingly glacial pace means it risks being eclipsed by global developments. The Bogota meeting is an opportunity for the EITI’s international board to show leadership and agree to bring the requirement for project-level reporting into effect. Otherwise oil and mining companies will be able to continue making payments in secret, and money that resource-rich countries badly need will continue being stolen.
By Miles Litvinoff, Publish What You Pay - United Kingdom
This post originally appeared on PublishWhatYouPay.org on January 3, 2017
Mandatory reporting by oil, gas and mining companies under European Union country-by-country disclosure laws began in the UK and France in 2016. Key aspects of the reporting requirements – which have equivalents in Norway, Canada and the USA – are especially useful in preventing corruption: granularity (disaggregation by project and by recipient government entity); comprehensiveness (all countries of operation without exemptions); and timeliness (the most recent financial year).
Eighty-six extractive companies reported under UK law in 2016UK-incorporated extractive companies must disclose payments within 11 months of each financial year-end, and London Stock Exchange-listed (Main Market) extractive companies within 6 months (unless the LSE is their secondary listing). UK-incorporated companies must provide open and machine-readable data. For most in-scope companies, the first reporting deadline was 30 June 2016 (if LSE-listed) or 30 November 2016 (if UK-incorporated and unlisted).
According to an assessment by PWYP UK and NRGI, by end-2016 disclosures of 2015 or 2015/16 payments to governments had been published by 86 UK-incorporated and/or LSE-listed oil, gas and mining companies (plus one forestry company). Prominent reporting companies include Anglo American, BG Group (now part of Royal Dutch Shell), BHP Billiton, BP, Cairn, Centrica, Gazprom, Glencore, Lukoil, Premier Oil, Randgold, Rio Tinto, Rosneft, Royal Dutch Shell, Soco, Total (main reporting obligation in France), Tullow and Vedanta. LSE-listed China Petroleum & Chemical (subsidiary of Chinese state-owned Sinopec Group) should have reported payments made in Angola and China but appears not to have.
Payments in which countries?Disclosures from the above named 18 companies provide data on 84 host countries. These include resource-rich developing and transition states where extractive revenues may be hidden or associated with the “resource curse”, and developed economies that also may fail to gain optimal outcomes from resource extraction (see graph and table of some of the countries where these 18 companies report).
How good is the reporting?This growing body of extractives data is essential – if not sufficient – to inform citizens, civil society, journalists and parliamentarians about the revenues generated by exploitation of their countries’ natural resources, how well the money compensates for negative social and environmental impacts and which government entities get paid (see PWYP’s Chain for Change).
The first year’s reporting in the UK needs improvement, however, and company disclosures are not always complete.
Difficulty in locating some reports and lack of open data
All UK-incorporated companies’ reports are available online in open data from Companies House, but there is no annual index; site users need to insert a blank space in the search box to produce a list of reports. LSE-listed companies’ disclosures currently lack a central location, making it hard to know how many have reported, and need not be in open data format. (Similar challenges occur with companies reporting in France.) LSE-listed companies are required to announce their report on the National Storage Mechanism but many do not, and none have used the correct classification. However, all LSE-listed companies will be required to upload their reports centrally in open data from 2018.
Over-aggregated or omitted data
Several companies have broadly – and geographically – aggregated data for multiple different oil and gas fields or mines: Shell (“Gulf of Mexico (West)”, “Northern North Sea”, “Sabah Inboard and Deepwater Oil”, “SPDC East”, “UK Offshore”); BHP Billiton (“Gulf of Mexico”); BP (“Gulf of Mexico Central”); Glencore (“DRC Copperbelt Region”). Very broad project aggregation may result in companies hiding suspect payments and arguably contravenes the law’s purpose.
Other companies fail to identify the government entities they pay, which PWYP considers a legal infringement. Lukoil lumps together payments to unnamed “state authorities”. Aggregate Industries (part of LafargeHolcim, which reports in France) identifies only unnamed “national” or “regional/local” governments. Petrofac initially failed to identify government entities but subsequently corrected this.
Companies are required to specify in-kind payments by value and volume and to explain how the value was determined. To verify price per barrel, value should be divisible by volume. However, Shell has for at least one project in Nigeria combined oil and gas in-kind payments in a single figure, making the price per barrel for each incalculable, and when requested refused to fully clarify. Petrofac originally combined cash and in-kind payments in Tunisia in a single uncheckable figure but then amended its report.
BP omits payments by non-subsidiary joint ventures (JVs), and Shell excludes payments by JVs over which it has joint control. Given the frequency of JVs in resource extraction, and because JV production entitlements are often the largest payment to a government, non-reporting of JV payments by non-operating partners – which could be reported proportionately – will leave large sums of money undisclosed.
Disclosures by Total and some other companies contain omissions of certain types of payments that require further investigation.
Civil society has been active in accessing and analysing the data, including via PWYP’s Data Extractors programme and PWYP US’s Extract a Fact site. Our work with the data will be the subject of future blogs.
The UK will review its regulations in 2017, followed by the European Commission’s EU-wide review in 2018. Civil society needs to engage with these processes to defend the value of mandatory reporting and, where possible, persuade policy makers to close loopholes and strengthen the law.
By Winnie Ngabiirwe, Global Rights Alert
Uganda is on the verge of an oil boom. At least 6.5 billion barrels have been discovered from less than 40% of the country’s oil regions making Uganda’s oil fields the third largest by reserves in sub-Saharan Africa. Once production starts, the revenues from oil are projected to substantially impact the domestic budget. If managed well, the expected oil revenues could transform the economy and dramatically improve living conditions for Uganda’s 37 million citizens, 13 million of whom live on less than $1.90 per day. Without proper management, oil revenues could instead exacerbate poverty by further perpetuating Uganda’s long history of endemic government corruption.
The government of Uganda has been mired by several high-level cases of corruption that deprive citizens of much needed funding for public services, and directly undermine billions of dollars in foreign aid Uganda receives annually. In fact, after one particular instance of embezzlement of funds in the Office of the Prime Minister in 2012, many major western donors temporarily suspended aid to the country. According to a 2013 report by Human Rights Watch, corruption has permeated all levels of government creating a destructive culture of impunity.
Seeking a means through which to address entrenched corruption in the country, the Civil Society Coalition on Oil & Gas in Uganda wrote in 2015 to the United States Securities and Exchange Commission. The group called on the agency to pass a strong rule mandating project-by-project disclosure from US-listed oil and mining companies that would bring much needed information to Ugandan citizens. The idea was enthusiastically welcomed considering that civil society has been largely unable to influence the Ugandan government to provide access to oil sector information despite several attempts.
Despite having a strong Access to Information Act in place, Ugandans lack access to almost all meaningful information regarding the country’s developing oil sector. While the government feigns interest in implementing various extractive industry governance initiatives, it consistently fails to respond to civil society campaigns for increased transparency. Most citizens have virtually no idea how much oil is in the country’s reserves, or how much revenue the government stands to receive from extraction. In fact, citizens’ most steady source of oil information comes from the local newspapers reporting on the latest scandal financed by pre-production oil payments.
Thus, civil society in Uganda enthusiastically welcomed the idea that more information would soon be made available to citizens through the passage of a strong US extractives transparency rule that required public reporting. While the US rule has stalled, the landmark law set a precedent that has been followed by 30 jurisdictions around the world such as the European Union (EU) and Canada. Despite delays with the SEC rule, Ugandan civil society has benefitted in the meantime from company disclosures filed in compliance with the EU disclosure requirements.
The government of Uganda has issued production licenses to Tullow, Total and China National Offshore Oil Corporation (CNOOC). Tullow and Total are both listed in EU markets and are publicly reporting payments under those UK and French laws. However, CNOOC is listed only on US and Chinese stock exchanges. This creates an uneven reporting regime among companies. The US transparency rule would have required reporting from CNOOC, expanding the availability of payment information for Ugandans about companies that will be operating in their country.
Since Uganda’s oil reserves are still in pre-production, civil society has had difficulty identifying the preliminary payments made by oil companies since these payments are even harder to track than production-based payments. The only major sources of publicly available information are the Bank of Uganda Annual Reports which provide incomplete and ad-hoc disclosure on the activities of the national petroleum fund.
However, with the EU disclosures published in 2016, civil society was able to examine the payment information reported by Tullow and Total and compare this information to the payments disclosed in the Bank of Uganda Annual Reports for fiscal years 2015 and 2016. This information has been used in direct dialogue with government officials as civil society representatives query discrepancies and demand financial accountability using real data, rather than hypothetical figures.
After reviewing Tullow and Total’s 2015 payment disclosures, civil society representatives found $14 million not included in government reports. Unless these funds were part of a prior transfer into the country’s general budget before the full operationalization of the Petroleum Fund, these $14 million in payments could reasonably be deemed to be missing. Equipped with this kind of information, civil society representatives have had much more valuable, in-depth debate with government officials to demand explanation for the missing funds. Civil society groups no longer have to rely on the political will of a government that has clearly demonstrated its disinterest in transparency. In this way, mandatory disclosure requirements on exchanges in major extractives markets are filling a very real void in Uganda and other resource-rich countries by providing information that would otherwise remain secret.
The relevance of this work was reinforced in January 2017 when it was revealed that Ugandan President Yoweri Museveni approved the payment of nearly $2 million in oil revenues to various government officials as a “reward” for their involvement in a successful lawsuit between the government of Uganda and Heritage Oil. This case was fought for four years, finally ending with a decision in favor of the government of Uganda over a disputed capital gains tax charge of $434 million. While the President’s decision demonstrates flagrant disregard for protocols set in the recently passed Public Finance Management Act, it also provides civil society monitors with important indications of areas of weakness in the current legal framework that could have been manipulated to allow for this sort of corruption.
Civil society can examine whether the Ugandan government is working under a questionable or unrealistic definition of “petroleum revenues” to artificially narrow the revenues that enter the petroleum fund. Alternatively, the statute outlining the permitted uses for petroleum revenues may be opportunistically misinterpreted to allow for corrupt payments. Or, the executive could simply be ignoring the law altogether.
These are the questions that will help civil society make real progress on effectively preventing future corruption in Uganda. Real progress must be based on real data so that civil society can meaningfully engage with Uganda’s secretive government in demanding answers. For the first time ever, newly available project-level disclosures have provided local civil society groups with the information necessary to query government, conduct investigations and demand accountability. Newfound access to payment information gives Ugandan citizens a much needed chance to inform development priorities and ensure that oil revenues lead to tangible positive outcomes for the citizens of the country. With access to this information, civil society is one step closer to opening Uganda’s opaque oil sector up to all citizens in the hopes that newspaper headlines no longer report on the latest government scandals, but instead on how oil revenues are being managed for the benefit of all rather than a few government elites.
Kathleen Brophy contributed to this piece.
Click here for the archives to see our full list of posts.