[Music] and I will hand it over to staff welcome great good morning uh mayor members of the council Mark Wittenberg I'm the Chief Information officer for the city and welcome we're going to be talking about artificial intelligence this is going to be an overview this is a really deep subject so I just want you to know this is going to be an overview of a little bit of an introduction of what artificial intelligence is what we're doing to prepare at the City of Raleigh some of our current and planned use cases around AI along with developing Technologies and Inn that we're seeing in the field and then what's next for us first a little bit of level setting so what is artificial intelligence first of all I want to start off with any of you trivia Buffs artificial intelligence has been around since 1956 actually started at a conference where several computer scientists got together and they coined the phrase artificial intelligence but you can think of artificial intelligence as like this big overarching subject with these other areas down below so at its core or at its foundation artificial intelligence is like a very smart assistant that can look at a vast amount of data perform calculations on that data and then provide results a good example of this is actually with our storm Water Division uh I was actually driving home the other day during the storm and I don't know if you've ever uh had this happen where the first block it's pouring rain you've got the wrench wipers are going full blast you go another block and it's not raining at all if we were to use our rain gauges which are actually dispersed throughout the city to predict the amount of rainfall and make decisions it actually wouldn't provide a clear picture so what we do is we use gauge adjusted rainfall or radar rainfall there I got that acronym right uh to be able to look at the radar and then make predictions and layer that along with the information we're receiving to provide a more accurate picture that we can then make decisions using modeling to understand what our rainfall is we use this for uh Public Safety for alerting the public and for uh predicting or how we're going to handle the rainfall the idea here is that I will just say Mitchell S councelor silver has an excused absence so I will state that for the record sorry to interrup absolutely no worries uh the next one is we're going to dive into machine learning so machine machine learning is taking that model of looking at a vast amount of data and then being able to train the model a good example here is intersection safety we have cameras at various intersections throughout the city and we are it used to be that we would count cars you ever drive over one of those cables in the in the street and it would count the car unfortunately that doesn't count bicyclists pedestrians it just counts the cars or you can stand at an intersection and physically count the vehicles what we were able to do is we were able to train the system to look at the video and to be able to understand the differences between like a truck a car a pedestrian or a bicyclist and then we were able to take it one step further and start to look at the lanes and understand churning movements the situation in the intersection Transportation takes all that information and now that instead of just a point in time they can look at this over a vast amount of time to be able to make intersection changes like adding a right turn lane to improve intersection safety the last piece is deep learning so deep learning is what you've all been hearing about in the news this is the generative AI so when you hear about chat GPT Gemini uh large language models that fits into the deep learning section of artificial intelligence for the remainder of the presentation this is what we're going to focus on because this is the latest the coolest thing that you're all he hearing about but I just wanted you to be aware that we've been doing artificial intelligence at the city for quite a while so what are the impacts uh Gartner who's a research firm is actually calling this like a new age or the age of cyber physical systems I don't even begin to like understand exactly what that means but for you all it means that it's transformative kind of like the Industrial Age we went from Steam Engines to combustion it's kind of the same thing we're going from the information age to this age where computers are just much faster and able to do much more because we have that capability or the speed so we're seeing impacts in productivity for example uh Lou actually asked me for an abstract for this presentation and he needed it by noon I took the presentation and all the backup material fed it into chat GPT and It produced the abstract that you see in your information I was able to do that very quickly but the key is to look it over make sure that you're you're the humans looking over that but we're seeing examples in productivity the job market just like any other uh Industrial Revolution the job market is going to have some impact uh in the jobs that we're doing we want to make sure that we're upskilling our employees and that we're providing training for those employees but there are definitely going to be impacts to the job market and we're seeing this transformational period I kind of like in artificial intelligence to what the calculator did to the slide rule right think of it as this new tool that's available to help you do your job better the economic development piece of it there is definitely opportunity in the Research Triangle we're seeing companies that are using artificial intelligence to be able to come to Market much quicker we're seeing opportunities in startups and entrepreneurs that are diving into this artificial intelligence and providing systems and then the last impact is uh environmental I mentioned that computers it's a lot of computing power uh saw a report that some of the big companies like Microsoft and Google are using anywhere from 30% to 50% more power and Microsoft is actually going to be purchasing some of the power from Three Mile Island and restarting Three Mile Island uh may sound a little scary but again they're trying to figure out how to be carbon neutral in providing this this computing power so what are some of the challenges and the risks and the threats around artificial intelligence well Bad actors when we think about Bad actors let's think about the good actors first we're all thinking how do we do this in an ethical and responsible manner think about the Bad actors do you think they're thinking about the same thing do you think that the criminals are thinking wow I need to make sure that I'm ethical about this probably not so the Bad actors are actually uh they're they're employing artificial intelligence much quicker than we can and then around the company AI use we need to be very cognizant what we feed into artificial intelligence what are some of the failures uh deep fake I don't know if you remember but during the governor's race uh one of the candidates there were a bunch of statements that the candidate had made somebody took and created a deep fake taking those statements and then creating that person actually saying those so it's kind of one of those weird things where the person said these things but the Deep fake was the person actually seeing a video of them saying these things and so I know there's legislature that is going through to talk about the political aspects of this and protections around Jeep deep fakes uh prompt Weare is is prompting AI to make it make mistakes or to teach it bad things because remember these models are learning that machine learning it's it's almost like telling it the the wrong things and then there's privacy There's No undo button on chat GPT if you feed something into chat GPT it is there and it is going to use that as data so if you feed it intellectual property or information about the city or infrastructure it now knows about that so we have to be very careful and make sure that there's go guard rails around that and then there's the uh unwanted bias is a big one we're going to talk a little bit more about this which is these models are basing their information on the data that's being provided and if that data is biased it's going to produce a biased answer for example uh I asked chat GPT to create a picture of a banker helping a client and what do you think that the banker turned out to be white male in a suit absolutely and then what are some of the results uh financial implications there's an example of an airline that had a chat GP like agent that provided some incorrect information to a client that resulted in a higher cost for that person's ticket they ended up taking the airline to court and the court cited in uh favor of the plaintiff which meant that they had to reimburse that person so the answers that these AI agents are providing are just as uh uh legally responsible as anything else like a human providing that answer and then human injury I don't think we'll do anything like this but there are examples out there where Google or ways take somebody on a dangerous route or if it doesn't know that a route is now uh no longer valid it will take them down a route potentially that could cause injury at the foundation of AI I mentioned data already once but that's really where it starts it starts around that security and privacy is we need to make sure that the data that we're feeding that's how it's getting those answers so we need to make sure that that data is secure and that we're also keeping that data private for example if we are looking at our infrastructure at our water infrastructure we want to make sure that we're not feeding that to an open model and that that infrastructure gets out there another thing is organization uh I was recently at the bookstore yes I still like books even though I'm a computer guy uh but it's interesting CU in the fiction area it's all arranged by author and in the non-fiction it's arranged by the subject same thing with chat GPT or any kind of AI technology is you have to arrange the data in such a way that it can get to it quickly and it can get to it accurately so we need to make sure that the library we can't we can't just throw it and say we're going to throw it at our website and let it peruse the whole website we need to make sure that we're organizing the data in a manner that the system can go through and do that quickly and then the Integrity of the data if the data is wrong it's going to produce a wrong answer uh if the data is uh biased like example there would be uh I inherited a large book collection from my grandmother and she was a big novelist kind of person so if somebody came and looked at my library they might make an assumption that I like whor instead of Science Fiction probably couldn't have guessed that one of the first things what's that the way you pronounced horror hor you can see now why I don't read those types of novels and that's why I stick with science fiction if you can't pronounce it don't read it right how does AI pronounce [Laughter] it uh the next thing we're working on is a policy uh the policy is around what are the dos and don'ts around the use of AI again we've been using AI for years and so this is nothing new but the generative AI is just a whole new area and because generative AI creates new things we there are some concerns there where before AI was just taking the data that we already had and making decisions and predictions based on that data now with generative AI it's coming up with new things so the risks we talked about some of the challenges and risks already and so we want to make sure that we're addressing those we've talked a little bit about the bias so in the policy we're making sure or we're we're asking that a human be involved in checking those answers do the answers make sense uh is it the answer that you want uh I hate to go back to my library example but that I like science fiction I'm not going to say the other word uh so making sure that those biases aren't there or that we're mitigating them as much as possible uh one of the examples there is going back to the banker example as I'm going through the presentation you'll notice some of the pictures are actually generated by chat GPT and there were several instances where I had to go back in and prompt it to uh be more diverse or take the third wheel off the car does weird things like that uh transparency is another big thing is if we're going to produce an answer and this goes back to college days which is if you're going to quote something you should site The Source same thing here is if we're going to provide an answer we should be providing where the source of that answer came from so you can go to the source and actually read the information information yourself and then we're transparent about the use of AI if we're using it uh and then accountability again going back to the airline example is we need to be accountable for the answers again that human is so important in the check and that's why the human is in the center of that and then the safety we need to ensure that any of the answers or use of AI that we have the community safety in mind I'm going to say it again just because I think it's so important the human is at the center of all this I remember my math teacher always telling me it's like solve the math equation but at the end look at it does the answer make sense and that's the same thing here is as we're starting to deploy these Technologies we really want to start internally and so we have somebody checking those answers to ensure that that accuracy the accountability transparency all the other pieces so what are some of the current use cases uh first one is Raleigh I don't know if you've all heard of Raleigh but that is our internal computer assistance for getting it help uh Raleigh is a chatbot and it allows you to ask it questions about like how do I get a laptop and it uses our database to be able to to formulate answers and provide those answers back so we've implemented a generative AI chatbot internally for internal employes and this is our way of testing out the technology before we were to deploy externally to our community uh one of the cool things about Raleigh is it started to learn very quickly and you where it started versus now you can ask it something or say something like my name has changed and it has actually learned based upon the things we've done for name changes to say oh if your name is changed you need to change it here and PeopleSoft you need to change it here and by the way do you need new business cards and would you like to order business cards so the idea is that it starts to learn what our processes are to be able to provide better answers uh web search is another thing we're testing internally first and this is that Search Assistant that you see at the right but you can ask things like uh how do I report a Miss trash trash collection and it will provide an answer with a number and then you can also see the find out more there's the reference to where it's pulling the information and where it's pulling the documentation uh but we've asked at things like you know what is the latest with brt or you can have it search the website and again it's it's that generative so it's learning and it's looking through our website couple of key points here with Raleigh we're pointing it at our internal it information with the web search we're specifically pointing it at our website so it's not pulling information from the rest of the world uh Public Utilities is another another great example Public Utilities was looking at the types of materials and failure rates for materials and water pipes that are in the ground so they contracted with the firm to pull all of your old records all the old Council records back as far as we could go along with purchase documentation and any other public information that we could find and aasted into a large database then we organize that database like the fiction non-fiction you arranging the the database and now we can start to ask it questions about the types of material that the water pipes and if we focus in and provide it data about failures of specific products we can now start to see where that might be in our infrastructure and make predictions of where there might be line breaks or future predictions of where we might have failures what we thought was really cool is the model was created for Public Utilities but then we asked it like who are the past seven Mayors and it answered the question correctly so we're thinking that we'll be able to pull that model in and provide a query for old public records for example like how much did the horses how much hay did the horses eat in the 1800s and we'd be able to pull that I don't know if we do that but but that's an example so just for the public what what is available to residents of Raleigh now and how do they access this so this is internal only because we're still testing internally so specifically these are all things that we're doing internally on the next slide I'm going to talk about what our plan use cases which is going to start to touch on the public uh and then the last one I'm sure you've all heard and I don't know it' be interesting to see how many of used chat GPT but we've deployed chat GPT teams and provided upskilling and training for employees and GPT is an amazing tool at writing uh for example like I mentioned the abstract creating uh uh pictures creating new things uh it's also fantastic for responding to emails and things like that but we're encouraging employees to use this tool in their day-to-day in productivity increased productivity so the planned use cases and this is what we're starting to look for in the future the building code search and council member silver isn't here but I was going to say the Udo right which is he loves acronyms right but the building code is quite complex so one of the things we're going to be starting to test is being able to use generative AI in front of the building code to start to ask questions like I'm building a fence what do I need to do and then it would query our Udo to provide that answer the next step would then be to start to step them through the permitting process so that's what we're starting to look at for the future is again that human piece of it which is providing this to the internal agents that are handling these calls checking the data before we release it to the public so this is just one use case that we're starting to to examine the second one you're going to be hearing more about customer service and the customer experience and the work that Karen Ray is doing around uh the customer experience and management uh for the Ser uh for City Services sorry got hung up on that one a little bit we're going to be initially rolling out in the June July time frame our online portal and that online portal is going to have the basic services and things that people do on a daily basis in the background AI is going to be used to start to look for patterns and what we're looking for are those other categories it's the categories where somebody hasn't found what they're looking for they found pothole or Miss trash can but there's something or some service that they want that they don't see there and so they write it out in text we'll be using AI in the background to start to look at those patterns to design workflows and then allow us to accelerate putting those onto the portal that's the first step the Second Step will be to enable generative AI where you'll be able to call chat social media email and all of that will go into a chatbot into the same generative AI so we're producing the same answers for that customer so the idea there is to provide those answers much more quickly if it's an informational it can search our database and provide that information right away without having the person wait for an agent or waiting for somebody from a staff member to call back so more to come on that and then Microsoft co-pilot is something else that we're looking at Microsoft co-pilot goes along with your Office Products so your Microsoft Word your PowerPoint it allows you to use generative AI to assist in the design and writing of documents checking your grammar which is very helpful for me your pronunciations even more helpful uh but it it's really like that assistant an example I use Microsoft co-pilot I had it last week so yes I am I'm using it uh last week I had summarize last week's emails prioritize those emails and then provide responses it took it about a minute and it did all that now you may ask well how does it know the priorities that's the machine learning part of it I said well no actually this one's a priority and if you see something from the city manager that's probably a top priority so it starts to learn what my priorities are and starts to figure out those to provide better answers so some developing Technologies uh the first one I want to touch on is video analytics I don't know if you all recognize this video but it is actually a building that is just next door we're flying this building with drones about every two to three weeks and we're documenting the construction proog progress as we go so the developing technology this is what we're doing today but in the future video analytics is going to get much better with the generative and the this deep learning piece of it for example we'd be able to do space planning and say based upon the video that you're seeing ingest this and what would be the best layout for an information technology department and it would provide that answer uh another example is video analytics and picture analytics where they're starting to head is ingesting a picture and then asking it where is the best place to hide and the generative technology will say well next to the stove but check to make sure it's not hot that's where we're heading with this is the ability to be able to analyze these types of pictures these types of videos and then start to ask it questions adaptive systems is the next thing right now when we think about these generative agents it's pulling on data that we have have and really that human component of checking the data where they're heading is agentic AI or these adaptive systems which has now imagined that the agent is self-contained and can start to make decisions on its own based upon the information that it's Gathering from past experience so uh the example here is looking at our our City downtown and the heat Island map what we'll be able to do is take a digital twin this which we already have and then based upon what we're building the trees we're planting and the things we're doing we can start to make future predictions about the climate so again taking something we have and then making predictions about the future and again this is a developing technology so this is looking this is where the industry is heading and what the future is the last thing Quantum Computing trying to figure out how to explain this so take the computer what we're doing today and imagine it's like 10 times faster maybe even more than that what is the computer speeds that we have today has allowed us to get where we are with generative technology and with artificial intelligence now imagine that computers in the near future are going to be very very fast so imagine the possibilities and imagine the computing power that we're going to have how far in the future Microsoft has just Rel released their first Quantum Computing chip and this is like in the last couple last month or so so this is really coming about very quickly but this is going to change the landscape of what you see so what we have today imagine it much much faster and much more available so what's next for the city of Ri so the first thing is we're really putting the finishing touches on that AI policy we're making sure that we have those guard rails in place and so as we start to deploy this technology that we're doing it in a very thoughtful and uh a very prescribed methodology the second thing is continued education really starting with you all here today which is just this highlevel overview of what we're doing again we can dive deep into any one of these various subjects also upskilling our employees we're already offering classes on chat GPT but we're going to be offering those co-pilot classes and then we really want to get together and start sharing these use cases where we see departments using Ai and how they're doing it we want to make sure that we're piloting those use cases and moving those use cases forward that we discussed and looking for other opportunities the last thing is we're really tracking that vendor Innovation so we're watching what's happening out there we're watching pure cities and we're watching those developments to see how those can be used here and with that uh myself chat GPT and co-pilot are here to answer any questions we have questions go for it I always have questions um hi thanks for this um I've been eagerly awaiting our AI policy um I'm curious will it come back to council for like the AI policy once it's finished it will require Council approval so typically the AI or a policy would go through the staff policy process and that's where it is it's just starting to enter the policy but mamson city manager yes it is an administrative policy but after we get to the point of implementation we would come back and do an an update and a refresh of what the policy entails how it's going to govern how we work as well as the governance structure for the policy got it okay um helpful um I'm curious about several things I've got several questions um first going back to Raleigh the internal Search tool um is someone quality testing it like is someone going a human going back through and saying these are the answers it provided to the questions and marking yes you are accurate so that it can continue to learn from its mistakes yes correct so both Ry and the web search tools have a log of the questions and then the answers that provided and we are going back and looking through those logs to make sure that uh the answers are correct the other piece of it is we're using those for our internal agents so that they can they can ask it questions and then review the answers to ensure that accuracy okay and that's that's really the big thing is is we want to make sure that we're internally testing these things first sure um absolutely and then you mentioned that it um for the web search you've got it the database pointed at the City of Raleigh website but we've been we've been told previously related to the website refresh there's like thousands and thousands of pages and some of them are relevant and there's probably archive pages that are not is it um able to access old pages and accidentally give a false answer so I'm going to go back to the uh security piece of it and I'm going to talk a little bit about that I'm also going to provide like a co-pilot example Le the data is so critical uh to this and so this is a a very good question I'm going to answer the web question but I'm going to form it in the like let's say a word document we were working on a Word document we're collaborating at the end of the day we abandoned this document and we started a new one finished that that's the one we published if that old document is out there and not marked as a draft AI will will find it if it has access to it and it will use that information to provide answers that's part of the data cleanup that we need to go about doing as part of the redesign of the website that is exactly one of the things that we're going to do is we're going to be looking for old outdated uh sites or sites that may have contradictory information and that's another area where AI will help us out quite a bit is we it will be able to start to identify those areas so we're going to be working very closely we it is going to be working very closely with Communications as we go through the website redesign um and then I will just ask a couple more and then I'll pass the mic and come back to my other questions if there's more time but um I work in customer service we have a chatbot um I would just offer I know you're you're not the lead on that but to Karen Ray or whomever um I would just offer for the sake of your agents to make it somewhat easy for people to ultimately get to the human if it's too hard or The Loop is too long people are irate by the time they reach the human and it's a a burnout and churn risk for your agents so just a heads up on that um and then you know one thing we know is that like the prompt you put in determines the outcome you get out right and um you said you're offering training but I wonder will you move to mandating training for um particular divisions of the of the city that are likely to use this particularly in like outward Focus like customer focused uses so I'm just thinking like if I put in uh you know help me write a response for this beep customer who's being so awful to me I'm GNA get one response versus if I write something like this customer is frustrated help me manage their frustrations you know they write content you put in determines the content you get out um like will you be how will the policy in put guard rails on like what people are putting in and then also like yeah are you going to move to mandating training for Relevant employees so the training is come a lot of the training is prompt engineering so you you definitely nailed it right there which is exactly what you ask it uh is the output you're going to get and sometimes you can lead the technology or at prompt where you may not get the answer you're exactly looking for co-pilot as we roll co- co-pilot out we're going to have a limited number of licenses and so we are considering a mandatory training like if you want to license for co-pilot you need to go through the training and part of the training would be the AI policy along with the policy we're also creating an AI guide book to kind of go along along with it to be more plain English like here are the dos and the don'ts and here's best practice so those are definitely things that we're considering pause thank you so much for all of this um as we go into and you're doing the policy uh work now how long from when you first started doing this to get to this point what does that time frame look like uh it's been about months to get to this point and part of that is uh Raleigh was a founding member in the AI Coalition which is a coalition of about 300 governments and 500 members and growing very quickly from State uh local and federal levels of government so as a collaborative we all work together to come up with like those guard rails and the things we should we should be doing so we were able to accelerate this very quickly based on our membership and the Coalition amazing I'm just trying to get a sense of how long these things take to do uh because it's such an a revolving technology and I I even I can't even get to the quantum that my brain doesn't even function there so I can't imagine how long that that would take so I'm just getting a snapshot um when you say guardrail I understand that you guys are creating them right now but teach me what a guardrail is and how what are you what are the things that you're looking for to make sure we're just an example you don't need to all of them because I'm sure there's a ton so one of the Guard reels be uh our own intellectual property making sure that somebody if they're using an open model like chat GPT which is open to the world that we're not feeding it proprietary data for example creating this presentation I said I am presenting to a city council a large city large city council on AI you know produce like this picture what I didn't want to do is I didn't want to feed it I work for the city of raw and here's my information so we need to be very cautious about that CU like I said there's no undo button so one of the guard rails would be around the information that you're feeding into the prompt got it got it and then last question for me as you look to implement it are you going to be prioritizing specific departments that you're focused on because it sounds like if we just open it up it's a lot of information for everybody um is there a targeted way like we're going to start with planning or or I don't know I'll leave that for you to answer so we have a governance committee uh that's formed that governs not only AI but many Technologies so our security policies and uh the way we spend our money on technology but that committee that it governance committee is responsible for the prioritization that committee is made up of members of the city manager's office along with every department head so as a group we're looking at like how can this best benefit the community when we start to prioritize we're really looking at at how can we add the most value and how can we do it in a safe manner that meets all the policy awesome thank you so much so you mentioned about proprietary information and not it not being used is there a way and are we able to tag any information that is being pulled um maybe where the server can tag it and say hey user ATS they get a message are you sure you want to use this information or it goes to a security team that says hey this was used by user y um just know as FY as a log so yes there are the water example that I provided is pulling data that's public record but it's in a closed model in other words it's in a model that's ours and in a in our in our closed environment so the rest of the world doesn't have access to it chat GPT doesn't have access to it just us for those types of models the guard rails are much wider because we don't have to be as worried about feeding it proprietary data because it's our data our model on our servers then with Microsoft co-pilot one of the switches is Microsoft co-pilot only looks at our internal data it doesn't go out to the outside world unless you toggle it there's actually going to be a toggle switch that you have to click Purp purposefully and then it will come up with a warning we're still working out those details but it will come up with a warning that says hey you are now entering information into an open model so you need to make sure that these guard rails are are to make sure you're not providing proprietary information other want to go back to um sorry more from me um one thing I know that the last time I attended like a conference where this was a topic that one thing that was unclear at that time is what aspects of prompts and outcomes would be matters of public record do you is there any news there yes and I might refer to the city attorneys but we've had multiple conversations with the city attorney's office this is providing or creating new content and providing it to the public which means it is absolutely a public record for example you can use the technology to take minutes from this meeting or actually record this meeting and then create a summary of this meeting with action items that becomes a public record because it is generating a new document that's why it's so important that that human really look over those minutes to make sure that they're accurate and then get posted like we would any other minutes or documentation of a meeting so is it only when something is sort of accepted like if it in this example if it generates the document and you give it a scan and you're like well that's garbage I'm going to have to make this myself and you sort of reject it is that is not considered to be a record I might have to defer here but I'm going to try to explain it and then the City attorney can either like shake your head yes or no but there's a difference between transient records and then the actual public record and so a transient record like if I were to take notes or Post-it notes and kind of like here are my notes from this meeting and I'm gathering all those notes together and then I create this final document the final document becomes the record these other things become transient records and they can be discarded and so the same would apply here is anything that's being used to create the final record could be discarded okay understood got it how did I do City attorney okay awesome and then um just two more um one is about the environmental impact um obviously lots of opportunities with AI but the environmental impact is really real it takes a lot of as you said a lot of computing power the data um data Resource Centers that are housing all this information that like server Farms are huge they use a lot of water and a lot of power and we have a climate action plan and I wonder how it's being discussed you know if we buy licenses for Microsoft co-pilot we could to theoretically be like well that's Microsoft's uh climate challenge but not ours but in the same way we are participating in the system so how how is the sort of climate impact conversation going uh to be honest we're just starting to have those conversations and invite and to figure out like how we're going to move forward but one of the things we're definitely looking at uh is how the power is generated and what these companies are doing so for example Microsoft seems to be more efficient than some of the other custom or some of the other companies that are employing AI the honestly I can't answer that fully because we have we're just starting down that road and just starting down the what are the impacts but that is definitely a consideration as we're as we're moving things forward okay good well I hope you'll keep it on the top of mind yeah thank you for this it's great to see I love this image um would love to have anyway uh the upside is great the down and and I'm glad we're in a Consortium of thinking through these I think about downsides and even if the City of Raleigh has protected I just think about all the right the sewer pipes the gas pipes the all the sort of infrastructure and the mapping of that and even if we really protect all that information all of the thousands of people who have sought permits and have these drawings themselves you know you're not as certain so how do we take a really big risk sort of view of this and protect everyone in this city from downside utilization of of some of these uh this information honestly that's why we're taking a very programmatic step forward is we are not we want to we don't want to be on the bleeding edge of this we want to be very cautious and we want to make sure that we have the policies in place to protect that data and to protect the community and make sure that we do have those guard rails in place and that's why we're doing it in this order and that's why we haven't released anything to the public yet is we want to make sure that before we do that that we've hit all those challenges we've hit all those risks and that honestly we have a plan around those if something were to happen that we understand this is what we would do we want to make sure that that's all in place but this is nothing new that before AI we had all of this data and we had to protect this data and that's why I had the bags under my eyes is because I don't sleep at night worrying about this type of thing and what would happen if so again I just want to assure you that the foundation of that data layer you notice that security was at that Foundation because everything that we do starts there and it starts with protecting the community starts with protecting the data thank you so I'm process processing all the information and I I heard you say you know you have to be careful what you put in is there a way to unlearn something as AI it depends on the AI if it's chat GPT no if it's our own internal AI yes and that's one of the reasons that as we deploy we're going to be deploying a model that we own so that if it were to produce an answer that is biased incorrect that we can go in correct the data correct the model and then move forward as opposed to an open model like chat GPT or Gemini we really don't have control over that we only have control over the data that we're providing it what we could do is if there is incorrect data or if chat GPT were producing an incorrect answer would be to give it more data so feed data to uh news media or create an article that would correct that information oh W so I I understand why that's could be scary for some because they can't erase that information they meaning AI can't erase that information we would just need to feed it more accurate information to counterbalance the incorrect data that it has correct okay all right thank you okay any other questions all right thank you so much for this thank you work looking forward to the next steps all right and with that the work session is adjourned [Music]