The Data.One site was revamped to now be Data.Gov.HK
On Wednesday the Hong Kong Government’s Office of the Government Chief Information Officer unveiled their revamp of their Public Sector Information (“PSI”) portal, taking pride in still making available its 3000 datasets for the public to use for both commerical and non-commercial purposes. Hong Kong’s open data enthusiasts familiar with the old site “Data.One” will appreciate the new “Data.Gov.HK” site’s easier navigation, improved functionality and categories. With this revamp, the government is demonstrating its continued support of availability of PSI and echoing the Financial Secretary’s 2014-2015 commitment to push all government bureaux and departments to:
“[make] all government information released for public consumption machine-readable in digital formats from next year onwards to provide more opportunities for the business sector.”
The revamp is a step towards this goal, identifying what departments submit what data, and its ease of navigation encourages access and use of existing government data sets, which is good. Through there are some snags and errors in the new site, this is expected of any website update. Surely government is reviewing the problems but as the site doesn’t support a dialogue or connection with the public, that is hard to tell.
While governments around the world are realizing greater policy review through scrutiny, supporting greater civic engagement, and realizing better efficiency by supporting Open Data, the government’s revamp demonstrates that the Hong Kong government is still just catching up with past trends to publish government information data. While there is no debate that government making data available “opens up new business opportunities” as well as “bring convenience to the public and benefit society as a whole”, the site runs short in its approach that it can merely publish a number of datasets without a view of data quality or update a site and that apps and benefits will just materialize. To realize similar benefits as seen by other governments, the HK government should work with open data enthusiasts and also adopt the same open data principles and standards of the international community. Open Data Hong Kong recommends the government adopt Open Data standards and principles, and reflect that with the Data.Gov.HK site. Namely: Adopt an open license on the datasets The Terms and Conditions are problematic for supporting app-building, collaboration, and analysis of government data. The following terms are examples:
“you shall reproduce and distribute the Data accurately, fairly and sufficiently;”
Data used for apps and analysis will get reformatted. And inaccurate data will have to be corrected and modified by developers and analysts. Would this be considered breaking this condition? It is unclear, ambiguous, and thus stifling. Although the terms and conditions include a waiver of responsibility by the government, this condition is confusing at the least and overbearing at the most. OGCIO should adopt an Open License such as CCO or copy the UK Open Government License on the datasets. Improve communication & engagement Users of the site can’t find out which datasets are new, if there are any. As much as the site would like to see use among the 3000+ datasets (with more to be added regularly), supporting better communication and engagement among users would be useful for information to flow both ways.
It’s not enough just to have datasets available, the data has to be effective, relevant, usable. How can users provide feedback about the datasets and communicate to government about the effectiveness (or uselessness) of a dataset? Or that a dataset is missing? This is not in the functionality of the site. An example of doing this comes from the Government of Canada where they ask for users to “Suggest a data set”, as well as functionality for users to provide a rating on datasets, and the hosting of open data competitions (which are also open to people outside of Canada). It’s about the data. Data Quality. Users of the datasets continue to have problems with the quality of the data. This hasn’t changed with the site revamp. Users have reported unclear and inexistent schemas for datasets, inferior data types for the data, and inexistent API support for pervasive connection with data. Although data sets can be very different from one another, the right file type and data connection goes a long way to supporting app development and value for the public, and avoiding severe headaches from app developers and head-scratching from analysts. We encourage OGCIO to provide departments or provide the right expertise and support for bureaux and departments making datasets available. Commitment to the future The data.gov.hk update demonstrates government’s support for making data available to support the public. Open Data Hong Kong continues to reach out to OGCIO and the rest of government to improve this, while recommending full support of open data to best realize public benefit.
The HK Budget is out. Open Data enthusiasts, engaged citizens and media are still reviewing the budget. The speech, figures, and now the data tell a story about what happened since last year and what is planned for this year. While others are looking between the lines, comparing totals from previous years, line spending of bureaux and departments, and new “sweeteners”, we are looking at the budget differently: at the format it is in and how everyone can better dive into it, better understand it, and have meaningful insight.
With the budget in a more open data-friendly format, more people can more easily review the budget, crunch the numbers, and get the answers they’re looking for. More is possible when people can get right into the budget without needing to scrape totals and dig up past budget totals and build spreadsheets and databases.
Last year the Financial Secretary’s Office (FSO) released the estimates figures as an Excel Spreadsheet file, and this year the estimates are released in a machine-readable CSV format. These are steps in the right direction to support a more accessible budget for better transparency and oversight of the budget. Open Data Hong Kong supports civic engagement and review of the budget, and part of our efforts is working with Code4HK on a “Hack the Budget Hackathon” for engaged citizen to apply their skills, interest and curiosity on the budget.
Few government documents have as much impact on the public as the government budget. It is important the budget is transparent, supports participation for decision making and policy making, for effective oversight and accountable review. The goalposts for these are changing. With rising levels of public capacity to review the work of government, technological tools to solve problems, and changing methods of engagement between government and the public, there are many opportunities for government budgets to report, engage and react:
Budget transparency – use open data formats to provide accurate quality detailed structured data;
Budget participation – remove barriers and conditions of use of the budget, and support fora (online and in-person) of public engagement of the budget;
Budget oversight – support mechanisms for timely updates, and means to address questions, policy recommendations, and review of government priorities and programs.
Governments around the world are moving in this direction, and so is Hong Kong. Projects like the Open Budget Survey rank and compare countries with links to budgets for comparison.
To better support public review of the budget, we submit the following 3 recommendations:
Unconditional: Remove any restrictive terms and conditions or licensing on budget data.
Structured: Support structured data for better quality data and greater detail of totals;
A structured data format (ie: provide a schema and provide data in XML or JSON format) is richer better quality data. You can see where the totals come from, how they total up, and what fields are linked from where.
Historic data: provide past budgets in machine-readable format
Past budget data helps to review trends and comparability. Digging these numbers from currently available information is tedious and unreliable, requiring authoritative and consistent figures.
Earlier this month we presented to FSO our set of recommendations ahead of a discussion to realise this. See the presentation material (has some updates). We will continue to follow-up and provide advice and direction to achieving these goals.
Eternal Sunshine of the Spotless Internet The Michel Gondry film “Eternal Sunshine of the Spotless Mind”, a Sci-Fi Romantic Comedy tackling the unpredictable consequences of erasing all memories of a love affair, is quite a good primer on the potential side-effects of the “Right to be Forgotten” (RTBF) policies, where online search engines can be compelled to “forget” information that is deemed outdated. As the internet is increasing becoming the host of our collective memories, moves to delete specific parts of our digital histories are likely to have equally painful and unintended outcomes. Our colleagues at Open Knowledge are clear that these moves pose a threat to transparency and open data, setting up a Personal Data and Privacy Working Group to discuss the matter. Currently a hot issue in Hong Kong, with the Hong Kong Privacy Commissioner Allan Chiang Yam Wang recently expressing strong support for the controversial policy (including on his blog), yesterday ODHK decided to delve further into the issue by hearing some local perspectives on the matter. After a bit of a break, the ODHK meets were back with a jam packed bill including presentations on RTBF by Frankie Chu of InMedia and economic governance activist David Webb.
Hong Kong InMedia is a citizen journalism advocacy organisation, and Frankie Chu started proceedings pointing out that for countries with poor access to information legislation such as those in Asia, any moves to alter search engine results will disproportionally effect citizens ability to access to information freely. Lacking archiving ordinance also makes Hong Kong particularly vulnerable, something which the Hong Kong Archives Action group has highlighted. In-Media has covered the topic regularly, and have put together a fantastic animation that gives a great overview of the potential risks of RTBF legislation, particularly from a Hong Kong perspective. Writing an open letter, In-Media and the Hong Kong Civil Liberties Union are also collecting signatures to let the Hong Kong Privacy Commissioner know public opinion is against these moves.
David Webb gave a presentation based on a recent talk he gave at the Christmas AGM of the Hong Kong Library Association covering a lot of similar ground, but going into more detail the test cases that have brought us to where we are today. Highlighting that Hong Kong media organisations such as the HK Standard are already reducing their archives, the most chilling consequences would be the growing political leverage that search engines like google, or J. Edgar Hoover types in security services and governments would have by becoming gatekeepers of information. David’s slides are available from his Webb-site here.
The Meet ended with a Q&A from Frankie and David, and updates and news from some of the ODHK working groups such as Guy Freeman and the OpenGov.HK access to information portal, and Scott Edmunds and the Open Science Working Group. Bastien plugged the ODHK and Code4HK “Hack the Hong Kong Budget” make on the 28th February, and the next ODHK meet will be on the 9th March, so watch this space for further news and updates. We look forward to seeing many of you there.
This post first appeared in Cantonese in the HK Economic Journal on 30 January.
A level-five typhoon hell-bent on wreaking havoc unexpectedly changes direction. Instead of moving towards Japan, as predicted, it’s now heading straight for Hong Kong. As soon as the typhoon’s new course has been noted by local meteorologists, your trusted Hong Kong Observatory app has bombarded your smartphone with alerts, warning you to take note that the level has been raised to nine – indicating black rain. You now know to take your laptop home and prepare to stay indoors the following day, as work is officially canceled. This, and many other innovations we take for granted, is made possible thanks to open data.
Open data is a relatively new concept in Asia and most governments in the region are wrestling with how to best include it in their national economic and technological strategies. With the exception of a few countries, most of the data Asian governments hold is closed, meaning users are limited in the reuse or analysis of valuable public information. In some cases, you might even be breaking the law by hosting government statistics on a site. When data becomes open, all this changes. Coined ‘a new goldmine’ by The Economist for its usefulness, open data has an estimated annual economic impact globally of at least USD $3 trillion.
Open data has the potential to unlock a bounty for Hong Kong that can be applied in a plethora of highly constructive ways, in addition to a significant economic impact. The socioeconomic benefits from liberating public data alone are substantial; the free flow of information and data reduces the friction that slows down development in both emerging and advanced internet economies like Hong Kong’s. More specifically, a country that fully embraces open data will see tangible gains in its creative, entrepreneurial, and innovative output – aspects crucial to sustaining Hong Kong’s long term prosperity.
According to the World Bank, Hong Kong currently ranks fourth in East Asia and Pacific and 18th in the world for its Knowledge Economy Index, ahead of Japan, Singapore, and Korea. This shows that the city is doing a commendable job to ensure its Internet environment is conducive for knowledge to be used effectively for economic development. Yet, quantitative rankings alone are not enough to fully portray a country’s open data preparedness and many facets of Hong Kong’s ICT sector requires attention to ensure users are well-equipped to convert data to gold.
Hong Kong has made tremendous progress in embracing open data and connecting hungry coders with a feast of public sector data sets. For example, the establishment of a Public Sector Information portal, Data.one, and in 2011 the Hong Kong government released selected data sets to the public provided by government departments. In addition, in 2011 and 2013, the government launched app competitions with the ICT community to encourage software developers and IT students to develop ideas and solutions based on the published data sets. The most recent competition yielded 100 contributions, of which 22% of the submissions were based on traffic, 16% on weather, and 12% on air pollution data sets.
While app competitions are a good way to encourage user participation with public data sets, Hong Kong is not actively engaging the community through a dynamic exchange between stakeholders from the supply and demand side. Creating a dynamic community for developers and end users to collaborate is essential for the sustainable development of a market’s Internet economy. Taiwan, for example, is one of the few countries in the Asia Pacific region to have successfully combined information push and pull elements in the open data environment, thus enabling a dynamic market for data. They have achieved this through aggressive community engagement and collaborations between the government and wider business and civil society, in addition to partnering with the UK open data Institute to bring about more liquid Internet economies.
On the regulatory side, Hong Kong faces some challenges. The city currently lacks comprehensive access to information laws as well as copyright regulations that would provide users a sound basis for business built on public data. Recognizing the need for a stronger regulatory environment to protect intellectual property and encourage innovation, the Law Reform Commission is currently reviewing a revision to the city’s information laws and the government recently proposed a series of initiatives under the theme of “Smarter Hong Kong, Smarter Living” in an update to the Digital 21 Strategy. As part of the new Digital 21 Strategy, the city will make, “all government information released for public consumption in machine-readable digital formats from next year onwards to provide more opportunities for the business sector.”
While these initiatives are progressive actions, the recent call from the Hong Kong Privacy Commissioner to extend the ‘right to be forgotten’ to Hong Kong and Asia threatens to reverse all our nascent progress on open data in Hong Kong. The right to be forgotten will open up holes on the internet and search engines in particular, hurting businesses’ access to this “marketplace of ideas” for data-driven innovation, and users’ access to information as well as right to know. This takes us in the wrong direction of progress; the destruction of information has an adverse affect on the betterment of our economy and overall society.
Open data is empowerment. The more access people have to public data sets, the more new ideas and innovative solutions tailored to the needs of society are developed, effectively unleashing a significant business impact while simultaneously elevating Hong Kong’s overall competitiveness. The wider business, ICT and academic communities are just starting to get involved in Hong Kong’s open data development, now is the time to galvanize an unobstructed flow of information for all. Waltraut Ritter is a member of Open Data Hong Kong, a managing partner and research director of Knowledge Dialogues, which she founded in Hong Kong in 1997, and a member of the government’s Digital 21 strategy advisory committee.
This month government, academics, and citizens (and Open Data Hong Kong!) joined the HK Government Efficiency Unit to share insights on the potential of better reporting and sharing of Air Quality monitoring data. Entitled “Open Data and Citizen Science Reporting Potential for Air Quality Monitoring”, the seminar was an exploratory event to see what could happens if we share what people are doing with air quality data, the challenges we face, and the potential ahead for pilot projects and more. Weather and pollution data are no brainer areas to open up to an Open Data approach, as they are topics of interest for concerned and engaged citizens, and the Hong Kong Observatory already makes much of this data available to the public through the AQHI (Air Quality Health Index) website. Connecting the producers of this data with downstream users, from government, academic and non-academic backgrounds, should help maximize the value of this precious data. Open Data Hong Kong are experienced and well placed to advocate as a collective voice for more data sets and better quality data on the environment and air quality, and we’d like to thank Kim Salkeld, the head of the efficiency unit, for inviting us.
Mart has already posted his quite detailed notes on the talks, but to summarize, the first part of the afternoon featured representatives from many of the relevant government departments like the Environmental Protection, Agriculture, Fisheries and Conservation, the HK Observatory, and the Efficiency Unit “1823” enquiries and complaints hotline. Ivy So from the BioDiversity Division of the Agriculture, Fisheries and Conservation Department presented on some of the interesting apps they’ve developed like “tree walks”, but most interesting from a citizen science perspective has been moves to allow the public to post pictures and register animal sitting on their HK biodiversity database and Eco map portal. They’ve also promised to release much of this data as XLS files, so watch this space to see the result. It would be great to have hackathons, visualisations and apps built using this data, and there is a shortage of useful biodiversity data in the global biodiversity GBIF databases, so anything to boost this is much needed. John Chan from the HK Observatory also covered some of the Citizen Science side, presenting on their engagement with schools and interested individuals through their Community Weather Observing Scheme. The second part of the afternoon presented to point of view of the public. This is where Open Data Hong Kong stepped in, Bastien from ODHK setting the scene with a short overview on the benefits of openness and transparency, and showing a few case studies on how open pollution data has been successful in countries such as the UK (see his slides here). Representing Code4HK Vincent Lau and Harry Ng gave civic hacker perspective, showing examples of what they and others are doing with this data (e.g. their real time visualizations of the AQHI data thrown together in one hour), and highlighting the shortcomings and difficulties working with the data in its current form. Andrew Leyden was the last speaker in the ODHK section, and also highlighted the potential and problems experienced by hands on users of HK Observatory data when building the “Hong Kong Air Pollution” App.
What was clear from this section reiterates the main issue with publicly accessible datasets in Hong Kong. They have great potential, but are presented wrongly, and under unhelpful restrictive licenses, so much time and effort is spent unnecessarily scraping, cleaning and processing this data, these datasets are legally and not practically interoperable with others, all of which reasons put off many potential users. Bastien showed Hong Kong’s ranking in the global open data index, where we are placed 13th in the world for open emissions data, and the main thing preventing our (70%) score from topping the table was the lack of true open licensing. This would be a very easy issue to fix, costing nothing, and massively increasing the potential utility and reuse of our data.
The final section of the workshop brought on some of the formal experts from the Environmental Protection Department and local academics working on this area. Zhi Ning from CityU and Alexis Lau from HKUST both presented data from mobile pollution detectors, CityU doing more grass roots and medium scale sensing work using open hardware arduinos and working with local schools. HKUST have been working with larger scale detection units built into trams, and they in particular have huge amounts of air quality and modelling data collected around the pearl river delta and beyond that would be fascinating to open up and let others work with. Currently hosting 400TB of data that is available for academic if not other purposes, it would be great to liberate and see what could be done with this data if it was made more widely available from public repositories.
There are plans to meet again, so watch this space. With the Efficiency Unit’s support, they can rally government departments around the table, and it would be great to set up a working group at ODHK to continue this work. Any interested individuals should contact us if they were interested in participating and helping us to advocate for open data to help understand and improve the local air quality.