OT: Grad School rant

I'm taking a Data Analysis class this semester for my two masters programs and I have to ask you GIS, Data entry, computer people HOW THE HELL DO YOU DO THIS EVERYDAY!!.

I spent 20 hours this week doing mapping and excel and PowerPoint work for social vulnerability data for a county and I'm ready to take a long walk off a short pier. Don't get me wrong the data itself is interesting and useful in public safety/emergency management but how can people enjoy inputting it and mapping and staring at a screen for hours at a time??

(Thanks for letting me vent after another long week)

Forums: 
DISCLAIMER: Forum topics may not have been written or edited by The Key Play staff.

Comments

There are a lot of people that can sit and data entry with minimal people-ing...

To you from failing hands we throw
The torch; be yours to hold it high.
@BuryHokie #ThanksFrank

how can people enjoy inputting it and mapping and staring at a screen for hours at a time?

It's getting automated away - scrapers, OCR, etc are all making manual data entry obsolete.

For most people in the data space, data entry/cleaning is far and away the worst part of their job. But it's a necessary evil.

Twitter me

Someone needs to let this professor know that then. Between 4 kids school and activities and sports, my birthday this week, school work, putting everything back after the hurricane scare and my daughters birthday last weekend I'm getting about 4-5 hours of sleep a night. Like I said I love the data itself and the databases we use to find a lot of the data are amazing but I can't look at another excel sheet, pivot table or PowerPoint but I have like 8 more weeks of this hell.

Wet stuff on the red stuff.

Join us in the Key Players Club

I mean, it's fully automated away yet... we're probably 10 years away from that. It would be like someone in 2003/4 having to memorize multiplication tables, even though everyone would actually be walking around with a calculator in their pocket by 2014.

By the way, what's the challenge around data entry? Might be able to help you speed it up, or at least point you in a different direction.

Twitter me

A lot of it just revolves around taking the data from the websites and databases we are using like https://cemhs.asu.edu/sheldus. Then taking out what we need putting in excel sheets, then building power points to "present it". A lot of the times the tables and graphs don't come over the PowerPoint the right way and XY information gets cut off for example. It's just frustrating.

Wet stuff on the red stuff.

Join us in the Key Players Club

Oh so the problem isn't really 'data entry' - eg; it's not like you have to manually type 1000 rows of data into excel; but rather, it's not copying into a powerpoint correctly?

Why not just format the table in excel, then take a screenshot, crop it, and paste it into powerpoint? That's fine for a presentation.

Twitter me

Manually entering between 200-300 things into excel each week. That's what I've been doing because that's how I learned to do it back in the early 2000's after high school. But I must be missing a step somewhere. Honestly I just need an 80 in the class and I'm fine. But it's a class half full of public safety folks who struggle and recent college grads who think it's easy.

Wet stuff on the red stuff.

Join us in the Key Players Club

Where do these 200-300 things come from? You gave me a website, but can you give me a specific example? might be able to help.

Twitter me

I was gonna ask the same question... there are also dummy data generators that are out there if you need to fabricate information, but if this is stuff that already exists this should be a copy/paste, download, or some other type of really fast data pull.

So for example how many tornadoes occur in Florida, what zip codes, census tracts, what's years they occur, what moths, injuries, deaths, dollars, dollars in todays rate taking into account inflation. Etc.

Wet stuff on the red stuff.

Join us in the Key Players Club

So do you have to read articles and manually enter the data into excel? Or the data in tables on websites? Where are you getting the data from? I'm looking for a specific webpage(s) where you get dates, injury counts, etc

Twitter me

So the data is already in the databases but he wants us to enter into our own excel sheets, then create PowerPoints using tables and graphs as if we were presenting it to leadership or elected officials.

Wet stuff on the red stuff.

Join us in the Key Players Club

Can't tell you the number of times I've prepared very easy to understand graphs/displays for elected officials and they still don't get it, or I really overprepare for what I anticipate to be contentious or potentially complex, then they sit silent with no questions and proceed to vote on something. Gotta love it!

there's no reason you should have to be doing manual entry if this is already in a db. There should be some mechanism to export that data to a .csv format and then you can avoid all the data entry entirely. All you should be worried about is boiling down the information so its at a level your intended audience can understand. As someone who has been on the presenting and receiving end of these I can say with 100% certainty you should not be posting raw spreadsheet data into the presentation. A sample maybe, but thats it.

"To learn how the systems work, how the data is derived" or so I've been told.

Wet stuff on the red stuff.

Join us in the Key Players Club

So the data is already in the databases but he wants us to enter into our own excel sheets

I'm 95% sure there's an easy way to export to excel. If you give me a link I can probably show you how.

then create PowerPoints using tables and graphs as if we were presenting it to leadership or elected officials.

You can also do this in such a way that the table stays in sync with Excel, and you can also lock the formatting. Or you can just make the diagram in excel, take a screenshot, and paste it into ppt.

Twitter me

You say it involves GIS data. There should be shape files or the data that you can query and export from the state information.

To you from failing hands we throw
The torch; be yours to hold it high.
@BuryHokie #ThanksFrank

soooo the trick with the X and Y axis getting cut off is that excel automatically defaults to 0 as the min end for each if there isn't a negative number. If you manually set the axis in the chart to a lower number (e.g., -1) you might be able to avoid having stuff get sliced off. Also, I know Microsoft loves to show how their apps integrate with one another by having a live excel sheet/chart in your ppt, but my advice would be to paste the stuff you copy from excal as a picture, and not a sheet/chart/object etc.

Also, I know Microsoft loves to show how their apps integrate with one another by having a live excel sheet/chart in your ppt, but my advice would be to paste the stuff you copy from excal as a picture, and not a sheet/chart/object etc.

I second this wholeheartedly.

No, I *don't* want to go to the SEC. Why do you ask?

We don't love dem Hoos.

So don't use the built in short cuts.

Wet stuff on the red stuff.

Join us in the Key Players Club

If you need to move an excel chart to PowerPoint just copy the chart from excel, then in PowerPoint click on the "Paste" drop-down and select the picture icon. This will literally convert the excel object or data you copied into an graphic. This makes it real easy to add effects and other edits on top of the graphic in the powerpoint

I'm sure the prof knows this already but they teach this stuff (btw, which course exactly is this?) so that you understand how the automation stuff works and you have the principle understanding to how it all works. I had a similar challenge with an analytics course ~2 years ago working with on the order of 100k data samples, and yeah - it gets overwhelming.

What I would advise for you to do immediately is get Microsoft PowerBI. Think of it as a visualization version of a pivot table. It can ingest massive amounts of data and help you make quicker sense of the data by giving you an easy way to develop visualizations.

Hazards Analysis and Disaster Planning for a masters in emergency and crisis management and a second masters in public administration.

Wet stuff on the red stuff.

Join us in the Key Players Club

I would second using Power BI. In addition to easy visualizations it also has the ability to manipulate data quickly. I work in digital/data consulting, so if you have any questions I'd be happy to help.

I second the Power BI recommendation, you can even use Power Automate to scrape data from the web and input that directly into Power BI for visualization purposes. If you don't know python for web scraping purposes this could be an easy way to do what you seem to be talking about.

And I turned twenty-one in prison doin' life without parole
No one could steer me right but Mama tried, Mama tried
Mama tried to raise me better, but her pleading, I denied
That leaves only me to blame 'cause Mama tried

Agreed. Or tableau. You should be able to get a student version for a year with any reputable university. Tableau let's you get tableau prep as well which is designed to do what I think you need. You can spend about 20 minutes watching YouTube videos and be able to merge all your data very easily.
I'm currently working on a census tract level merged data set project with Places Health Outcomes and Social Determinants of Health. The very first thing I did was just merge everything through tableau, add the shape file and start visualizing before I headed over to python to start manipulating. Sounds like python would probably be beyond the scope of your project but I highly recommend spending a few hours developing a basic level of powerbi or tableau. Pretty solid chance you will get well over an 80.

Ring Design Chair

Totally OT but our band is playing Palencia's Fall Festival on Nov 4; 5-8 pm. Very family friendly event and just down the road from you guys.

Bourbon. Copious amounts of bourbon.

If you play it, they will win.

"How the ass pocket will be used, I do not know. Alls I know is, the ass pocket will be used." -The BoD

Tedious day in the data mines? Bourbon

Dealing with idiots and idiot bosses who don't know shit and think they know it all? Bourbon

A good productive day doing cool shit with maps? Celebratory bourbon

Yep, there's a reason this particular ass pocket is in the field.

If you play it, they will win.

"How the ass pocket will be used, I do not know. Alls I know is, the ass pocket will be used." -The BoD

Still in a shitty situation with your bosses?

I've been in web server hell trying to build hosted web services and web maps and apps for underground utilitiy field maps that are locked down enough to not get me in trouble with the feds.

Unfortunately, that ain't changing. I'm still here, but I ain't exactly happy about it. My damned work ethic is a SOB because I am deadset on finishing some things I've started before I get serious about jumping ship. That, and I can throw that shit on the ol' resume. Plus, I just can't bring myself to do that to my technician without getting him trained up more. Not trying to do what my former boss did to me before he retired.

Let me know how the temp is in web server hell. I'm about to pivot to laser-focusing on everything web-based very soon.

If you play it, they will win.

"How the ass pocket will be used, I do not know. Alls I know is, the ass pocket will be used." -The BoD

Don't set up a web server if you don't have to. It's actually not that hard on the ESRI side, if you have somebody on the IT end to help you through all the firewall obstacles and IIS hoops. But I only have our water & sewer utilities on there for security reasons. Even though the AWS/AGOL cloud is probably more secure, if the managers found out I had that data on the cloud they'd freak. Everything else is on our ArcGIS Online site and I update with scheduled models in ArcPro. ArcPro made it so you can host everything on AGOL without having to update manually.

It's already set up. I need to do some cleanup on it and AGOL. We have a bunch of maps and apps that were made years ago for different departments and scenarios and most of them haven't been touched since I started here. So many departments have a penchant for being shown things they can do with GIS, getting really excited about it for all of about a week, and then they never touch it again. I'm going to get rid of a bunch that are obsolete and tweak some others.

I hear you on the security issues. We got a new crime analyst a while back and after finding out it was compliant decided on the AGOL route for her and the maps she makes for the sheriff's office.

I wish I had her in my dept. She came in having only used QGIS, but all I had to do was show her Pro and AGOL...I'm talking didn't have to do much more than point at them and give her login credentials... and she took it and ran. If she was in our GIS dept I am fairly certain my biggest worry for the day would be making sure the coffee is hot and ready.

If you play it, they will win.

"How the ass pocket will be used, I do not know. Alls I know is, the ass pocket will be used." -The BoD

Oh yeah, I had to clean up a bunch of that stuff when I got here in 2015. I'm to the point now I need to go back and update a bunch of my own web apps with the new templates they've created.

Glad to hear you're totally set up though. Once you have all that in place and your map services organized it can get fun.

it can get fun.

/S 🤣

If you play it, they will win.

"How the ass pocket will be used, I do not know. Alls I know is, the ass pocket will be used." -The BoD

I see "Data Entry" and "Computer People" as comprising a pretty large spectrum

Data Entry is it's own special kind of hell. Separate and distinct from all the other types of computer hells.

Yea, I've never met anyone who does 'data entry' for a job. I've met people who create content/data, and I know a lot of people who have to 'clean' data as part of their job. But no one who's sole job is data entry.

Twitter me

To an extent, manual exception payment processing.

From the 2018 VT-uva game-"This is when LEGENDS are made!"

I don't do Data Entry/Verification anymore, but I did get a great deal of satisfaction out of it when I was doing it.

There was a visible result at the end of the day to see how productive I had been during the day.

Never Forget #1 Overall Seed UVA 54, #64 UMBC 74

Been in the mortgage company I work for for thirty years next March. Never had a lot of college courses in MIS or data stuff in college (VT from '83-'88- I missed IBM punch cards by one year and was the last class to not HAVE to have my own 'PC"-which remember was essentially lugging around the equivalent of a desktop tower with a 4" amber monitor and still using 5 1/4" floppies). When I first started part of my duties was compiling monthly sales/production reports which they were doing in Lotus123 and had constants that had to be manually zeroed out each month and new data entered. Average time each month was 5 full working days to complete. I converted it to an excel spreadsheet with monthly tabs and a tab for YTD all of which calculated with formulas so all you needed to do was enter current month figures and the rest was automated. Completion time dropped to two hours. Big sense of accomplishments.

With ya on the 'presentation' side of things. I hate making things 'look pretty'; my value is in seeing patterns and anomalies in data (what fits here and what doesn't make sense? and does X only become a problem when Y happens?). My current position is as a liaison between the IT folks who know software/hardware but often don't have a real clue what data-fields actually represents (just 'widgets' to them) and the business folks who know the inputs but don't know how the system/computers work. Having done most every job in the mortgage business besides default and accounting, AND the same company for so long, I have a unique knowledgebase that can help find the details behind the numbers. Kind of a 'corporate memory' that can't truly be replicated. 10-11 more years 'til I can retire with 40 years service and lucky that the recently merged company restarted an actual pension, so every additional year I work adds $100 a month in pension income.

From the 2018 VT-uva game-"This is when LEGENDS are made!"

I found out quickly at my first job out of college that one of my skills is a high tolerance for tedium. Kind of like 07Hokie, I knew I was going to have a visible result at the end, so that was the pay off I was looking for. For me, that pay off was a cool looking and informative map that wowed people.

But to commiserate with you, getting there with the data is frustrating. I cut my teeth in the GIS lab in Major Williams Hall. I honestly think that's what caused my hair loss to accelerate. One day Bill Carstensen was in there helping us and we were asking a question about importing and appending data and he just said, "At some point you have to get creative with your problem solving and just figure this shit out." At that point I figured out the point of the labs and the projects wasn't to go through the steps and learn what the steps taught you. It was to get to the end result by any means necessary.

I've been in 911/municipal gov't GIS for like 15 years now and the hardest and most frequent thing I've done and still do is take data from a source with a spatial element in the table and get it on a map.

I started a fiverr gig doing general GIS work for anybody who needs it and is willing to pay. My most frequent customers there are adults in college and rich college kids who need a project done. For some reason it's always Canadian kids. I did do a routing project for a landscaping company in DC who bought a bunch of customers from another company and wanted to balance and optimize their workload. But I'm not advertising my gig though. I'll just help you if you need it.

Municipal GIS checking in. I learned to accept the tedium when every task involves citywide data and manual review. I can't tell you how many fishnets I've made to methodically go through data.

What's a fishnet, and why haven't I heard of it, and how do you do it?

Also, I see you graduated right after I did in 07. We probably had some classes together.

Shoot me an email at hsorlp AT yahoo. com I'll reach out to you. Thanks

Wet stuff on the red stuff.

Join us in the Key Players Club

I just got back to you. I didn't see your message until just now.

I love it!! It's like putting together a puzzle. And then when you get it all figured out, then it's like an investigation to see what the data can tell you. And don't even get me started on all the possibilities with AI when you have a nice clean data set. I seriously wish I had discovered and studied the field when I was in college

"The Big Ten is always using excuses to cancel games with us. First Wisconsin. Then Wisconsin. After that, Wisconsin. The subsequent cancellation with Wisconsin comes to mind too. Now Penn State. What's next? Wisconsin?" -HorseOnATreadmill

My people!

I read "Data Analysis" and got all nerd hyped hahaha

"The Big Ten is always using excuses to cancel games with us. First Wisconsin. Then Wisconsin. After that, Wisconsin. The subsequent cancellation with Wisconsin comes to mind too. Now Penn State. What's next? Wisconsin?" -HorseOnATreadmill

Yeah, I read data analysis and GIS and went from 6 to midnight.

You're doing an MPA right? I finished my MPA last year and when I got to stats/data analysis courses I had much of the same feeling. However by the end of it, I really came to appreciate it more. I was never a big stats person but came out of it with a better perspective on it as a tool for policy analysis. But agreed, it was major stress going through it a few years ago.

How does anyone work in an office, for a job, for life? Or even any extended amount of time? I did it for one year and despite busting my ass, being successful at it, hated the bullshit corporate environment and found myself staring out the window wishing I was outside. That was plenty for the rest of my lifetime.

You will see this game, this upset and this sign next on ESPN Sportscenter. Virginia Tech 31 Miami 7

His decision was made after a phone call with longtime Virginia Tech assistant coach Bud Foster. All Foster told him was, "We win. They don't."

See APFOW entry above. Bourbon is a pretty good start.

Money

Yeah, it literally pays the bills. I think for most of us out there you hope to find a job that challenges you mentally and pays enough to live the life you want to live. I'm lucky to have that. Is it my dream every day? Of course not, but I like what I do enough and it keeps the kids fed and clothed with enough leftover for beer and Hokie tickets

I should be glad that enough people are willing to settle for the money, so everyone isn't out there trying to do what I do, but I just can't wrap my head around it. Ironically for me, my job wasn't secure and didn't come with a bigger paycheck working in an office for someone else, as society said it would. I have numerous best friends who spend all day, week and year in office buildings, apparently making lots of money. Life is too short for that, in my opinion. I'll take the a more interesting typical day, tax benefits and lifestyle of being my own boss. He who dies with the most stories wins.

*edit: I'll also add that it does seem like a lot more people have a better work / life balance now due to Covid and get to spend more time at home than before.

You will see this game, this upset and this sign next on ESPN Sportscenter. Virginia Tech 31 Miami 7

His decision was made after a phone call with longtime Virginia Tech assistant coach Bud Foster. All Foster told him was, "We win. They don't."

Yeah I grew up what many would consider "poor" so I am more than content to spend 40 hours babysitting a computer (from my own home now thankfully) in exchange for the funds so I don't have to worry about bills getting paid and whether or not a flat tire, leaky pipe or any other random small thing ruins a holiday or the next several months. I don't need to love my job so long as it doesn't drive me crazy and allows me to afford the things I do enjoy outside of working hours.

This is why the GIS field was appealing to me. I do all the boring computer stuff but I still have to go outside and GPS stuff and these days I fly our drone to do aerial topo surveys for the engineers to design off of and update aerial imagery. It's a good mix of inside and outside work.

Which is why I went with Civil site versus mechanical when choosing long ago. Rarely the same day over and over...lots of variety.

To you from failing hands we throw
The torch; be yours to hold it high.
@BuryHokie #ThanksFrank

I go talk to local college kids pretty frequently and that's what I tell them is the biggest upside. Inside/outside and something different almost every day.

I tell them it's different ish, same day....

To you from failing hands we throw
The torch; be yours to hold it high.
@BuryHokie #ThanksFrank

Lately I've been riding my motorcycle to work for an hour, wishing I was on my motorcycle for the next few hours then riding my motorcycle home for an hour.

I got to ride down to Asheville, NC for 7+ hours then attend class for 3 days and return home same 7+ hours.
It was wonderful.

I had to drive a car today. :(

This is going to be great for the ACC.

Now that's an awesome way to commute. This spring I sold my street legal Yamaha WR250R to a friend so he could ride to and from work. Please be careful, I already rescued a man from certain death this summer after he crashed his motorcycle speeding and have heard of four more bad wrecks since then, including one death. Almost saw a truck hit a motorcyclist today on the road actually.

You will see this game, this upset and this sign next on ESPN Sportscenter. Virginia Tech 31 Miami 7

His decision was made after a phone call with longtime Virginia Tech assistant coach Bud Foster. All Foster told him was, "We win. They don't."

sadly (for me personally), for the last 50 years almost all of the public policy development and implementation that "saves" the outdoors, occurs in offices

Also, for my fellow data scientists, if you have remote opportunities or know of opportunities in Dallas, I'm finishing up my dual degree MBA/MSBA in a December and would love to work with fellow Hokies.

Ring Design Chair

So I have so many projects I want to do with data analysis, and the largest hurdle is always how do I find a set of data I can scrap for free. Excel does have features to pull in adat from websites that's works okay-ish. But normally I just write some python code to scrap the web. Then I put the code into mysql and deal with it safely. Well as safely as SQL allows.

GIS, Data entry, computer people

This is a wide group of people lol. Personally I enjoy solving difficult problems without a clear defined "answer", but yes staring at a screen can be draining. It's important to develop strategies like frequent walk/snack breaks to keep your mind fresh.

How I Learned to Stop Worrying and Love the Jet Sweep

As someone who works in banking.....that's what entry-level Analysts are for.

I feel every word of this my brother (former hosedragger / current grad student / day job cloud engineer) . My best mitigation strategy is do this kind of mindnumbing work for a bit, get the muscle memory established and then put on a podcast or lecture and try to increase the efficiency of the time somewhat. It doesn't make it suck any less, it just makes me feel a little better about it. Strength and courage to you, it's an ass-kicker.

Oh man. Saw this and was hoping it would include grad school rants for lab work? I could go on for days on that topic! To this day, I consider getting out of academic bench science and into market research business consulting 1 of the 3 best decisions I ever consciously made in my adult life.

Alas, large volumes of data entry and cleaning and computer work were not required when I was trying to understand how nasopharyngeal carcinoma cells become resistant to chemotherapeutic drugs (grad school) or trying to develop an assay system for rapid screening of HBV medications (post doc). But for anyone suffering through any type of grad school, I feel for you.

Recovering scientist working in business consulting

The problem is the tools you are using. I want to kill myself after using excel for more than 10 minutes. Ask me to crunch data with Python and D3 and I'll work 7 days straight and it'll feel like ten minutes.

Fyi, if you want to go from data to visual pretty quickly, check out:. https://observablehq.com/@d3/gallery

Current civil engineer with professional license, and former software engineer & aerospace engineer; what keeps me going is the end result. It's why I'm on my third career, which is also the most satisfying. What I have discovered about myself is that I want to make a good living, but I also need my work to be useful and beneficial to our society. That's what keeps me going for hours on end in front of my monitors. I work in our water utilities group and as someone who used an outhouse in his first 6 years of life, I find great enjoyment in my projects replacing old water pipes with new ones.

Go Hokies!

I was required to take two 100 level GIS classes for my degree. Interesting stuff and I use it more now than what the degree is actually for, but holy shit was it tedious and time consuming. This coming from a notoriously bad student.

Amateur superstar and idiot extraordinaire.

GIS from 10 years ago or so versus today's version is night and day.

To you from failing hands we throw
The torch; be yours to hold it high.
@BuryHokie #ThanksFrank