AI Policy Making: Week 1 Summary
AI Policy Making
Week 1 Summary
Midjourney prompt: a native NZ kiwi bird, wearing Google Glasses, in New Zealand native bush, photo realistic --ar 7:4
The Goal
Having been using ChatGPT for many Resource Management Planning tasks since February, I was curious to see how well it would perform drafting RMA policy. The RMA system has been getting exceedingly complex, and with the upcoming RMA reform, it is likely to get even more so. The existing system is getting confusing for even experienced practitioners, so how well would AI work in such a complex regulatory framework?
Rather than experimenting with 'theoretical' scenarios, wanted to use a real district plan review as a baseline comparison. Preferably one that is already doing the review process so that the data and technical reports were current, and if recent community consultation had been undertaken, that would be a bonus.
… Hello Kaipara!
As with every council and every district plan, The Kaipara District Council seem to have been doing its District Plan FOREVER. It started its District Plan review in 2020. According to the latest District Plan Briefing Project Update (May 2023) some elected members are concerned about even continuing with the review in light of the upcoming RMA reform.
It seemed like an ideal time to kick off this project. Council workshops to examine individual topics had already been held between November 2022 - June 2023. Two full plan briefings are scheduled for July and August, with the request for approval to notify going to Council in September 2023. An Alternative Bootleg District Plan could be run in parallel, and the results compared.
The Artifact
I wanted to build something real - an actual District Plan. In this way I could test both the product and the process. By using Kaipara as a case study, I could also follow their communication channels to their customers to test their views as the Bootleg District Plan evolved.
Those with a tech startup or service design background will recognise the lean and agile approaches I have adapted for this project. In my experience many of the tools and methodologies for developing computer code are readily transferrable to developing legal code. I have adopted the 'lean startup' business model to build a 'minimum viable product' (MVP) District Plan. I wanted a minimalistic District Plan that contains only those things that are legally required to be in a district plan, and then add to it only where it created value for the user.
The MVP District Plan
Section 75 of the RMA prescribes the contents of district plans. It says:
A District Plan “must state”:
Objectives
Policies
Rules (if any)
A District Plan “must give effect to”:
National Policy Statements
NZ Coastal Policy Statement
National Planning Standards
Regional Policy Statements
A District Plan “must not be inconsistent with”:
Water Conservation Orders
Specified Regional Plan matters
Design Parameters
The RMA has a few other provisions that set out the processes to be followed when preparing a District Plan and the matters that must be considered along the way. The MVP District Plan would need to tick all these boxes, so the first stage of plan development would be to build a 'wire-frame' to define the scope and structure of the plan, then populate with more detailed content, starting with the objectives and policies. The next stage would be to evaluate the policies (s32 cost/benefit analysis) using an adapted lean canvass method to develop rules or other methods to achieve the objectives.
Publishing the MVP District Plan as it is developed enables customer feedback during the plan formation process. In a previous project I discovered that adopting a hyper transparent approach was one of the key reasons for the project's success. The community could see the authenticity in the process, and were more willing to engage for that reason. Because we really didn't know what the end result would be, and whether the project would even work, added to the level of interest.
Defining Scope and Structure
Asked Bing ('Creative' mode) to describe those matters that the RMA says MUST be included in a District Plan. It confirmed:
"Therefore, if you want to create a minimalistic district plan, you should focus on developing objectives, policies, and rules that are aligned with higher level planning documents and legal requirements. You should also avoid adding any unnecessary or redundant information that does not add value or clarity to your district plan. You should aim for a district plan that is concise, coherent, and consistent."
“When I asked Bing in ‘Balanced’ mode to provide a list of documents it refused to give one and said to contact the Council of MfE directly for more information. Bing on ‘Precise’ mode I found to return a less extensive list than it did in ‘Creative’ mode, so be sure to check the results you get.”
So far so good. But I wanted to test its accuracy by giving me a list of all the documents that direct what must be included in the Kaipara District Plan. Bing replied with an accurate list of "some of the documents", but only provided others when asked for the rest. To check that Bing was able to access the content of these documents rather than just the titles, I asked:
"Let's then start with the National Policy Statements. List all of them from oldest to newest, followed by any draft NPS currently under preparation. For each, I want a list of things the Kaipara District plan must give effect to."
"Now a list of regional planning instruments and iwi management plans that the district plan must not be inconsistent with. Oldest to newest including drafts. Under each, list the matters that the Kaipara District plan should or should not include."
Bing listed each of the national and regional documents required to be considered, with bullet point summaries of the matters that the Kaipara District Plan must give effect to, and matters that should be consistent with. With the broad scope of the Plan established, the structure of the Plan could be laid out:
"Based on the l national, regional, and rohe matters listed above, give me a list of topics / chapters for the Kaipara District Plan that we can set up and begin to examine one at a time. This is for the purpose of setting out the topic and compiling information we know about each topic. Part of this exercise will involve my collecting published data and generating maps. After that is done, no doubt patterns will emerge, and topics may overlap or begin to relate to some areas, and affect some communities more than others. The topics may evolve accordingly, but for now, let's start with a list of the main topics / chapter headings so we can begin to work on objectives."
The initial list of Chapters was:
Introduction
Tangata Whenua
Strategic Direction
Natural Environment
Built Environment
Checking for Bias (Legacy Influences)
I wanted to explore whether the 'Natural' and 'Built' Environment distinction was derived from the RMA reform process presently under way, or whether it was a legacy 'rural' vs 'urban' distinction that is found in most legacy plans (and perhaps AI training material bias). I asked Bing to consider other ways of slicing topics such as on a catchment basis, with upper, middle, and lower catchment slices. It took those aspects into consideration and noted that this method could also address cross-boundary issues and cumulative effects within and between the bands, helping achieve more integrated and effective resource management.
To get a better idea of how the vast array of resource management topics would be allocated to which Chapter, I asked for subtopics for each of the Chapters.
Introduction
Te Ao Māori
Te Tiriti o Waitangi
Ngā Iwi me ngā Hapū
Ngā Mahi Tahi
Ngā Kaupapa Here
Strategic Direction
Vision and Outcomes
Integrated Management
Sustainable Development
Growth Management
Key Issues and Challenges
Natural Environment
Land
Water
Air
Biodiversity
Coastal Environment
Built Environment
Infrastructure
Transport
Energy
Waste
From this step it was evident that many of the more common District Plan provisions did not appear in the list, such as people, communities, housing, open space and recreation, community facilities, education, culture and heritage. I noticed that the structure seemed to be heavily influenced by the National and Regional Policy Documents, and there isn't any mandatory policy direction in relation to the missing topics. An instruction was therefore sent to Bing to take into account the RMA purpose and principles.
“I hadn’t included that prompt earlier as to those of us working in the resource management system it is a given - be sure not to make such assumptions!”
The Limitations of Bing
Bing in 'Creative' and 'Precise' modes have a 4000 character limit per prompt and a 30 prompt limit per conversation. Bing in 'Balanced' mode has the same total prompt limit, but only a 2000 character limit per prompt. ChatGPT-4 on the other had has a much longer conversation 'memory', so it was at this point that I switched platforms. A potential work around, which is the one I used to get ChatGPT up to speed, is to copy the whole conversation into a website (or document hosted on the cloud) that could be pointed to in the first prompt of a new conversation. If you do this with Bing, just remember that its three different 'personalities' have varying degrees of comfort when providing the ‘correct’ answer. I’ll talk about this further in a future post.
Cross Prompting to ChatGPT
Having already uploaded the conversation with Bing into a blog article, I tested whether ChatGPT-4 (with Bing search enabled) could read the background a continue with the conversation. It worked much better than I expected, and included very current topics that do not appear in many current District Plans. I prompted a check against the RMA Planning Standards (which prescribe District Plan formats), and ChatGPT-4 seemed to have a pretty good understanding of how the Plan needs to fit within those Standards. It noted that when specific rules and other provisions are developed, they will need to be added in.
This approach is consistent with the MVP instruction I gave at the outset, and my preference for a 'lean' approach to developing policy and methods, rather than starting with an existing palate of colours (zones) to paint onto the planning maps. I was conscious not to influence or bias ChatGPT with templated solutions at this point, but also wanted to compare this result with the one that the humans at Kaipara District Council has come up with.
I prompted for a comparison between them both, which provided a very useful check for quality control purposes. ChatGPT suggested at that point that we could merge the lists, but I reasserted my preference for a minimalist MVP version at this time to avoid any bias towards status quo when we do the s32 cost benefit (lean canvas) evaluation of the policies later in the plan development process.
The ChatGPT response was far more thorough and insightful than I had expected, including references and accurate application of RMA provisions that had not featured in the conversation at all until this point. It also provided accurate responses to my prompts asking it to distinguish between RMA directions on how things are done (process) and what things are considered (scope), but left out of the planning document itself.
In the words of ChatGPT, the MVP approach "aligns with current trends in planning, which are moving towards more visually-oriented and spatially-specific plans. This kind of approach can make it easier for the public to understand the plan and how it applies to their property or neighbourhood." With things all looking positive and compliant so far, it was time to start adding layers to the wire frame.
Establishing a Content Outline
My next step was to flesh out the Table of Contents with a content outline for the Topics and Subtopics within each Chapter. The initial reason for doing this was to get some visibility around which topics went where so I could check for gaps and overlaps. It turns out that this was one of THE most critical steps I took, as being able to point back to the content outline enabled me to get ChatGPT back on track whenever it got confused. I’ll show how I did that in a future post.
Once the document outline is completed, the generating content for the Strategic Direction Chapter was very rapid. But, I also discovered how important it was to maintain the flow of the conversation and not change direction mid topic - for several reasons, which I will explain in a future post.
Testing Approaches ‘On the Fly’
One of the most interesting things about using AI is its ability to critique your work as you go. Several times in the conversation (ask between Chapters, remember!) I threw in some of my personal theories and approaches to policy development based on previous experience. By asking what ChatGPT thought about my techniques, it provided useful feedback on the strengths and potential weaknesses of each approach.
Note however, that the choice of ‘personality’ you want the AI to adopt can give very different results in this respect! More on that later.
Getting into the Details
By the end of week 1 we had completed the District Plan Structure, full content outline, and populated the Strategic Direction Chapter. Much of this direction setting has already been done by the Kaipara District Council, and the Strategic Direction draws heavily on the Council’s Vision developed in accordance with previous Local Goverment Act (LGA) processes and the directions set by the RMA and National Policy instruments.
So how well can the AI team of ChatGPT and Bing tackle some of the more geography specific issues? Let’s find out …