Hacker News Reader

Comment by danielbln | Hacker News Reader

I feel like I live in an alternate universe to this comment. Every engineer I know (devops, software, hell even datascience) is using GPT-4 or Copilot (especially Copilot with the new chat interface) and people, myself included, use it every day. Refactor this code for me, add a function that does X, why is Y failing, align this code with a different coding pattern, I cobbled together this React app, clean it up and refactor it and factor Z functionality into a node backend, help me understand this bit of obtuse Python code, and the list goes on and on and on. I've seen tremendous value, and its not without fault mind you, LLM hallucination is a real problem, but I would be very sad and annoyed if tomorrow they'd take LLMs out of my daily engineering flow.

View on HN

Copilot shines with a number of things and isn't very good at a number of other things. Whether you get value out of Copilot says more, I think, about the kind of work you do on a day-to-day basis than it does about the utility of Copilot.

Copilot/GPT is excellent at writing lots of new lines of code. It's also really good at getting you started in code/frameworks that you don't really understand.

However, Copilot/GPT is not nearly as good at troubleshooting problems in existing code. If your job involves lots of bug fixing or tweaks to existing features, Copilot and GPT are next to useless.

I've noticed that if my work falls into the first category, Copilot often speeds me up something like 30% to 40%. If my work falls into the second category, it's 0%.

How about this? Taking a dense obtusely written ML algorithm with a ton of mathematical notation, plug it into GPT4, and get torch code? Cuz I do that often.

That falls pretty clearly into the first bucket the parent listed, "Help write lots of code"

It's not better then me.

But I use it when I'm lazy, tired or at times drinking.

And then it's really nice. At times it costs me too much time solving it's mistakes. But then it nails something perfectly and I'm impressed again.

I often work with new (to me) tech or build prototypes/PoCs, it's pretty great for that.

It's useful to recognize when a dialog is a dead-end, otherwise it's easy to enter a rabbit hole that the llm can't get out of. Best to poke at it from another angle and/or within a new dialog, then.

> It's not better then me.

Where did anyone say it was?

It’s normal. Early adopters always feel this way. It happened with home PCs, internet connections, the iPod, smartphones.

They will talk down to you about it until the wave rolls over them. It’s a tradition as old as mainframe engineers mocking PDP-11 hackers.

This one will be particularly rough for the late adopters, since one of the effects of ML code generation is compression of experience level.

Forgive the long reply, but I'm going to go ahead and "press X to doubt," because I have been trying to use both ChatGPT and Copilot, and the results are abysmal apart from anything but "write me a linked list" or "give me some React boilerplate." If you have examples of it working for you, please share them. Here's a very salient example of it not working for me: I spent about 1-2 days figuring this out, so it's definitely non-trivial, but it's not something insane either. I'm working on a new project/startup idea and this is part of it (verbatim prompt I just used on chatGPT):

    hi, can you write me a function in rust that, if running on windows:

    1. figures out if the topmost window is a windows explorer window
    2. if it's a windows explorer window, it returns the path (as a string) that the windows explorer window is actively viewing
    3. if it's not a windows explorer window, it returns a blank string

This is a deceptively difficult task, as it requires all kinds of COM finagling, and ChatGPT gets it royally wrong:

    #[cfg(windows)]
    fn get_active_explorer_path() -> String {
        use std::ptr;
        use winapi::um::winuser::{EnumWindows, GetWindowTextW, IsWindowVisible};
        use winapi::shared::windef::HWND;
        use std::os::raw::c_int;
        use std::ffi::OsString;
        use std::os::windows::ffi::OsStringExt;

        let mut path = String::new();
        let mut active_explorer = None;

        unsafe extern "system" fn enum_windows_callback(hwnd: HWND, lparam: LPARAM) -> c_int {
            let active_explorer = &mut *(lparam as *mut Option<HWND>);
            let mut buffer: [u16; 512] = [0; 512];
            if IsWindowVisible(hwnd) != 0 && GetWindowTextW(hwnd, buffer.as_mut_ptr(), 512) != 0 {
                let window_title = OsString::from_wide(&buffer[..]).to_string_lossy();
                if window_title == "File Explorer" {
                    *active_explorer = Some(hwnd);
                    return 0; // Stop enumeration
                }
            }
            1 // Continue enumeration
        }

        unsafe {
            EnumWindows(Some(enum_windows_callback), &mut active_explorer as *mut _ as LPARAM);
        }

        if let Some(hwnd) = active_explorer {
            let mut buffer: [u16; 512] = [0; 512];
            if GetWindowTextW(hwnd, buffer.as_mut_ptr(), 512) != 0 {
                let window_title = OsString::from_wide(&buffer[..]).to_string_lossy();
                if window_title == "File Explorer" {
                    let shell32 = winapi::um::shellapi::GetShellWindow();
                    if shell32 != hwnd {
                        let mut path_buffer: [u16; 512] = [0; 512];
                        if winapi::um::shellapi::SHGetPathFromIDListW(ptr::null_mut(), path_buffer.as_mut_ptr()) != 0 {
                            let path_str = OsString::from_wide(&path_buffer[..]).to_string_lossy();
                            path = path_str.into_owned();
                        }
                    }
                }
            }
        }

        path
    }

I'm actually on my Mac right now, and I don't even need to test it, because the first red flag is `"window_title == "File Explorer"`. File Explorer will change its window titlebar to the currently viewed folder (so the conditional will always fail). There's also other comical mistakes, including passing a null pointer to `SHGetPathFromIDListW`, using `GetShellWindow` which returns the desktop, and not taking into account that Explorer windows can have multiple tabs. Okay, let's see if we fare better on MacOS.

    #[cfg(target_os = "macos")]
    fn get_active_finder_path() -> String {
        use std::ptr;
        use objc::runtime::{Class, Object};
        use objc::{class, msg_send, sel, sel_impl};
        use objc_id::Id;

        let mut path = String::new();

        unsafe {
            let workspace: Id<Object> = msg_send![class!(NSWorkspace), sharedWorkspace];
            let front_window: Id<Object> = msg_send![workspace, frontmostApplication];
            let front_window_pid: u32 = msg_send![front_window, processIdentifier];
            let running_apps: Id<Object> = msg_send![workspace, runningApplications];
            let app_count: usize = msg_send![running_apps, count];
            for i in 0..app_count {
                let app: Id<Object> = msg_send![running_apps, objectAtIndex: i];
                let app_pid: u32 = msg_send![app, processIdentifier];
                if app_pid == front_window_pid {
                    let bundle_url: Id<Object> = msg_send![app, bundleURL];
                    let bundle_path: *const Object = msg_send![bundle_url, path];
                    let path_str: *const i8 = msg_send![bundle_path, UTF8String];
                    if !path_str.is_null() {
                        let c_str = std::ffi::CStr::from_ptr(path_str);
                        path = c_str.to_string_lossy().to_string();
                    }
                    break;
                }
            }
        }

        path
    }

    #[cfg(not(target_os = "macos"))]
    fn get_active_finder_path() -> String {
        String::new()
    }

    fn main() {
        let active_finder_path = get_active_finder_path();
        println!("Active Finder Path: {}", active_finder_path);
    }

Doesn't look like it: this is what I got, which doesn't even compile. I'm not as familiar with MacOS systems programming, so I'm not even sure what's wrong with it, frankly. My solution involved using osascript on MacOS (which is the first Google hit, though I'll have to port it, as some folks may not have it on their systems).

So what is ML code generation useful for? Apart from helping on the Nth CRUD app someone's building, I just don't see it.

When I have use cases similar to this one (similar in complexity), I'll pair program with it. If the code fails, I'll tell it what error I got and ask it to refactor. If the explanation it gives with the code shows it didn't understand the question, I'll guide it along the correct track.

It's not perfect at "zero-shot" answers but from my experience is very good when you work with it conversationally.

I get a lot of mileage out of GPT, so let me see if I can explain. I wouldn't ask GPT/Copilot to do what you asked. After using them for a while you start to get an intuitive sense of what's easy and what's hard for them to do, and your specific example is indeed too difficult to get GPT to do properly. They have the biggest utility for me for stuff that I would ask a junior programmer to do - like a new grad.

The real value proposition of GPT isn't that it can solve really hard problems. The value proposition is that it's about as capable as a junior engineer, except it can write code much faster than any junior engineer, so it can speed you along on the boilerplate-y parts of coding that otherwise would be a lot of manual effort. It's especially useful for things that feel "easy" but which I don't have the relevant domain expertise. For instance, the other day I needed to write a fairly trivial shell script to parse some JSON files. I never write shell scripts and I always forget the syntax, but GPT wrote it correctly on the first try. That probably saved me 20-30 minutes of googling for how to do things like read files, etc in shell scripts.

It’s great at simple tedious stuff.

It’s useful for average things and concepts we might be unfamiliar with.

It’s of limited use, if not utterly useless, for most things a senior would have a hard time wrapping their head around.

And where it really shines, is when adapting an answer you’d otherwise look for on Google to your code, context, and constraints.

Thanks for including this example. It helps me understand what you mean.

ChatGPT is not really equipped to do this level of code design because there are too many steps for it to handle at once. It can handle your instructions if it could first write a detailed design spec, then write code and unit tests, then run them, read the compiler output, and iteratively make changes.

IOW, this would require a purpose-built solution which uses ChatGPT's underlying engine to take your list of requirements and turn them into built and tested code.

> ChatGPT is not really equipped to do this level of code design because there are too many steps for it to handle at once

There's no code design going on here. My solution was literally just going through like 100 StackOverflow answers & Microsoft's god-awful documentation to get the 50 lines of code that does what I need it to do.

In fact, this is precisely what I'd hope ChatGPT would be good for. Most of my final code is simply copy-pasted from SO/example repos/official docs. He's the correct function (comments added by me so I'll know wtf this is meant to do when I look at it again in 3 months).

    fn get_context_path() -> Result<String, Error> {
        unsafe {
            // Init COM libraries in this thread
            CoInitializeEx(Some(ptr::null()), COINIT_APARTMENTTHREADED | COINIT_DISABLE_OLE1DDE)?;

            // CabinetWClass is the Explorer window class
            let class = s!("CabinetWClass");
            let topmost_explorer = FindWindowA(class, None);
            let foreground_window = GetForegroundWindow();

            // Breadcrumbs IFF topmost window is an explorer window
            if topmost_explorer == foreground_window {
                let windows: IShellWindows = CoCreateInstance(&ShellWindows, None, CLSCTX_LOCAL_SERVER)?;
                let unk_enum = windows._NewEnum()?;
                let enum_variant: IEnumVARIANT = unk_enum.cast::<IEnumVARIANT>()?;

                // Iterate through all IShellWindows
                loop {
                    let mut fetched = 0;
                    let mut var: [VARIANT; 1] = [VARIANT::default(); 1];
                    let hr = enum_variant.Next(&mut var, &mut fetched);

                    // No more windows?
                    if hr == S_FALSE || fetched == 0 {
                        break;
                    }

                    // Not an IDispatch interface?
                    if var[0].Anonymous.Anonymous.vt != VT_DISPATCH {
                        continue;
                    }

                    // We should be able to turn an IShellWindow into an IShellBrowser
                    let shell_browser: IShellBrowser = IUnknown_QueryService(
                        var[0]
                            .Anonymous
                            .Anonymous
                            .Anonymous
                            .pdispVal
                            .as_ref()
                            .unwrap(),
                        &SID_STopLevelBrowser,
                    )?;
                    let shell_window = shell_browser.GetWindow()?;

                    // NOTE: The window we're matching with is actually the shell's parent
                    // Given that windows can be tabbed, we want to make sure the parent is the top
                    if GetParent(shell_window) == topmost_explorer {
                        let shell_view = shell_browser.QueryActiveShellView()?;

                        // Do some COM finagling, including a QueryInterface cast,
                        // until we get the folder path we're looking at
                        let folder_view: IFolderView = shell_view.cast::<IFolderView>()?;
                        let folder: IPersistFolder2 = folder_view.GetFolder::<IPersistFolder2>()?;
                        let curr_folder = folder.GetCurFolder()?;

                        // NOTE: This fails on non SYSPATH paths (e.g. "Home" or "Recent")
                        let path = SHGetNameFromIDList(curr_folder, SIGDN_FILESYSPATH)?;
                        let result = path.to_string()?;

                        // To figure out what path we're on (e.g. what tab is active), we try to match the
                        // parent title with the folder
                        // FIXME: Unless using full paths as titles, we can have ambiguities here
                        let mut title: [u16; 512] = [0; 512];
                        let len = GetWindowTextW(topmost_explorer, &mut title);
                        let title = String::from_utf16_lossy(&title[..len as usize]);

                        // We found the proper tab (child)
                        if result.contains(&title) {
                            // Cleanup COM & return path result
                            CoUninitialize();
                            return Ok(result);
                        // Must be another one (or something went wrong)
                        } else {
                            continue;
                        }
                    }
                }
            }
            // Cleanup COM & fallback to blank string
            CoUninitialize();
            return Ok(String::from(""));
        }
    }

I've added the statement "let's think through the problem step by step" after your instructions. Here's GPT-4's answer. I don't know Rust so let me know if this is a better answer:

---

Sure, let's break down the problem:

1. Identify the topmost window: We can use the `winapi` crate in Rust to interact with the Windows API. The function `GetForegroundWindow` can be used to get a handle to the topmost window.

2. Check if the window is a Windows Explorer window: We can use the `GetClassNameW` function to get the class name of the window. If the window is a Windows Explorer window, the class name should be `CabinetWClass`.

3. Get the path of the active folder in the Windows Explorer window: This is a bit more complex. We can use the `SHGetPathFromIDListW` function from the Shell API to get the path from an item ID list. However, getting the item ID list for the active folder in a Windows Explorer window is not straightforward. We might need to use COM interfaces like `IShellWindows` to get the `IWebBrowserApp` for the window, and then use `IWebBrowserApp::LocationURL` to get the URL of the active folder.

4. Return the path as a string or a blank string: This is straightforward. We just need to convert the path to a string and return it, or return a blank string if the window is not a Windows Explorer window.

Here is a skeleton of the function in Rust:

  ```rust
  extern crate winapi;

  use std::ptr;
  use std::ffi::OsString;
  use std::os::windows::ffi::OsStringExt;
  use winapi::um::winuser::{GetForegroundWindow, 
  GetClassNameW};
  use winapi::um::shellapi::SHGetPathFromIDListW;
  use winapi::shared::minwindef::MAX_PATH;

  fn get_active_folder_path() -> String {
      // Get the topmost window
      let hwnd = unsafe { GetForegroundWindow() };

      // Get the class name of the window
      let mut class_name = vec![0u16; MAX_PATH + 1];
      unsafe {
          GetClassNameW(hwnd, class_name.as_mut_ptr(), class_name.len() as i32);
      }

    // Convert the class name to a string
    let class_name = OsString::from_wide(&class_name)
          .to_string_lossy()
          .into_owned();

      // Check if the window is a Windows Explorer window
      if class_name.starts_with("CabinetWClass") {
          // TODO: Get the item ID list for the active folder in the Windows Explorer window
          // TODO: Use SHGetPathFromIDListW to get the path from the item ID list
          // TODO: Convert the path to a string and return it
      }

      // If the window is not a Windows Explorer window, return a blank string
      "".to_string()
  }
  ```

Please note that this is a skeleton of the function and it's not complete. The part to get the item ID list for the active folder in a Windows Explorer window and get the path from the item ID list is complex and requires using COM interfaces, which is not straightforward in Rust. You might need to use a crate like `winrt` to interact with COM interfaces in Rust.

---

I don't write Rust but does this look (more) correct?

    extern crate winapi;

    use std::ptr::null_mut;
     use winapi::shared::guiddef::{CLSID, IID};
    use winapi::um::combaseapi::{CoInitialize, CoCreateInstance};
    use winapi::um::shobjidl_core::CLSID_ShellWindows;
    use winapi::um::winuser::GetForegroundWindow;
    use winapi::Interface;
    use winapi::shared::winerror::{HRESULT, SUCCEEDED};
    use winapi::um::exdisp::IWebBrowser2;
    use winapi::um::unknwnbase::IUnknown;
    use winapi::um::oleidl::IDispatch;
    use winapi::shared::wtypes::BSTR;
    use winapi::um::winnt::LPCWSTR;
    use winapi::shared::ntdef::VOID;

use std::ffi::OsString; use std::os::windows::ffi::OsStringExt;

     // Helper function to convert a wide string to a Rust string
     wide_to_string(wide: &[u16]) -> String {
    OsString::from_wide(&wide)
        .to_string_lossy()
        .into_owned()

}

    // Helper function to convert BSTR to Rust String
    fn bstr_to_string(bstr: BSTR) -> String {
    unsafe {
        let length = winapi::um::oleauto::SysStringLen(bstr) as usize;
        let slice = std::slice::from_raw_parts(bstr, length);
        wide_to_string(slice)
    }

}

   fn main() {
    unsafe {
        // Initialize COM
        let hr = CoInitialize(null_mut());

        if SUCCEEDED(hr) {
            let mut shell_windows_ptr: *mut IUnknown = null_mut();
            let shell_windows_iid: IID = IShellWindows::uuidof();
            let shell_windows_clsid: CLSID = CLSID_ShellWindows;

            // Create IShellWindows instance
            let hr = CoCreateInstance(
                &shell_windows_clsid,
                null_mut(),
                winapi::um::combaseapi::CLSCTX_ALL,
                &shell_windows_iid,
                &mut shell_windows_ptr as *mut _ as _,
            );

            if SUCCEEDED(hr) {
                let shell_windows_dispatch: &IDispatch = &*(shell_windows_ptr as *mut IDispatch);

                let hwnd = GetForegroundWindow() as LPCWSTR;

                // Get Item of IShellWindows
                let mut result = null_mut();
                let mut params = winapi::um::oaidl::DISPPARAMS {
                    cArgs: 1,
                    cNamedArgs: 0,
                    rgvarg: &mut winapi::um::oaidl::VARIANTARG {
                        n1: winapi::um::oaidl::VARIANTARG_0 {
                            n2: winapi::um::oaidl::VARIANTARG_0_0 {
                                vt: winapi::um::oaidl::VT_I4 as u16,
                                wReserved1: 0,
                                wReserved2: 0,
                                wReserved3: 0,
                                n3: winapi::um::oaidl::VARIANTARG_0_0_0 {
                                    lVal: hwnd as _,
                                }
                            }
                        }
                    },
                    rgdispidNamedArgs: null_mut(),
                };

                shell_windows_dispatch.Invoke(
                    winapi::um::exdisp::DISPID_ISHELLWINDOWS_ITEM,
                    &winapi::shared::guiddef::IID_NULL,
                    winapi::um::winnls::LOCALE_USER_DEFAULT,
                    winapi::um::oaidl::DISPATCH_METHOD,
                    &mut params,
                    &mut result,
                    null_mut(),
                    null_mut(),
                );

                // Convert result to IWebBrowser2
                let web_browser2: &IWebBrowser2 = &*(result.pdispVal as *mut IWebBrowser2);

                // Get LocationURL property of IWebBrowser2
                let mut location_url: BSTR = null_mut();
                web_browser2.get_LocationURL(&mut location_url);

                let path = bstr_to_string(location_url);

                println!("Path: {}", path);
            } else {
                println!("Failed to create IShellWindows instance.");
            }
        } else {
            println!("Failed to initialize COM.");
        }
    }
    extern crate winapi;

use std::ptr::null_mut; use winapi::shared::guiddef::{CLSID, IID}; use winapi::um::combaseapi::{CoInitialize, CoCreateInstance}; use winapi::um::shobjidl_core::CLSID_ShellWindows; use winapi::um::winuser::GetForegroundWindow; use winapi::Interface; use winapi::shared::winerror::{HRESULT, SUCCEEDED}; use winapi::um::exdisp::IWebBrowser2; use winapi::um::unknwnbase::IUnknown; use winapi::um::oleidl::IDispatch; use winapi::shared::wtypes::BSTR; use winapi::um::winnt::LPCWSTR; use winapi::shared::ntdef::VOID; use std::ffi::OsString; use std::os::windows::ffi::OsStringExt;

// Helper function to convert a wide string to a Rust string fn wide_to_string(wide: &[u16]) -> String { OsString::from_wide(&wide) .to_string_lossy() .into_owned() }

// Helper function to convert BSTR to Rust String fn bstr_to_string(bstr: BSTR) -> String { unsafe { let length = winapi::um::oleauto::SysStringLen(bstr) as usize; let slice = std::slice::from_raw_parts(bstr, length); wide_to_string(slice) } }

fn main() { unsafe { // Initialize COM let hr = CoInitialize(null_mut());

        if SUCCEEDED(hr) {
            let mut shell_windows_ptr: *mut IUnknown = null_mut();
            let shell_windows_iid: IID = IShellWindows::uuidof();
            let shell_windows_clsid: CLSID = CLSID_ShellWindows;

            // Create IShellWindows instance
            let hr = CoCreateInstance(
                &shell_windows_clsid,
                null_mut(),
                winapi::um::combaseapi::CLSCTX_ALL,
                &shell_windows_iid,
                &mut shell_windows_ptr as *mut _ as _,
            );

            if SUCCEEDED(hr) {
                let shell_windows_dispatch: &IDispatch = &*(shell_windows_ptr as *mut IDispatch);

                let hwnd = GetForegroundWindow() as LPCWSTR;

                // Get Item of IShellWindows
                let mut result = null_mut();
                let mut params = winapi::um::oaidl::DISPPARAMS {
                    cArgs: 1,
                    cNamedArgs: 0,
                    rgvarg: &mut winapi::um::oaidl::VARIANTARG {
                        n1: winapi::um::oaidl::VARIANTARG_0 {
                            n2: winapi::um::oaidl::VARIANTARG_0_0 {
                                vt: winapi::um::oaidl::VT_I4 as u16,
                                wReserved1: 0,
                                wReserved2: 0,
                                wReserved3: 0,
                                n3: winapi::um::oaidl::VARIANTARG_0_0_0 {
                                    lVal: hwnd as _,
                                }
                            }
                        }
                    },
                    rgdispidNamedArgs: null_mut(),
                };

                shell_windows_dispatch.Invoke(
                    winapi::um::exdisp::DISPID_ISHELLWINDOWS_ITEM,
                    &winapi::shared::guiddef::IID_NULL,
                    winapi::um::winnls::LOCALE_USER_DEFAULT,
                    winapi::um::oaidl::DISPATCH_METHOD,
                    &mut params,
                    &mut result,
                    null_mut(),
                    null_mut(),
                );

                // Convert result to IWebBrowser2
                let web_browser2: &IWebBrowser2 = &*(result.pdispVal as *mut IWebBrowser2);

                // Get LocationURL property of IWebBrowser2
                let mut location_url: BSTR = null_mut();
                web_browser2.get_LocationURL(&mut location_url);

                let path = bstr_to_string(location_url);

                println!("Path: {}", path);
            } else {
                println!("Failed to create IShellWindows instance.");
            }
        } else {
            println!("Failed to initialize COM.");
        }
    }
   }
 }

The problem is that you are using it for a "deceptively difficult task" and not "a task I know how to do, or could easily look up, but would prefer not to". Looking at my ChatGPT history, some things I've used it for in the last week (all GPT-4):

* Formatting a huge confusing blob of python code, markdown, and json into something readable.

* Writing "boring" python functions needed to interface with AWS DynamoDB & S3.

* Going from 0 -> 1 on a side project creating printable QR codes that redirect to custom links with logging. (Literally gave me picture perfect step by step instructions here, as if it was reading off the AWS UI. The sole issue I had it helped me troubleshoot).

* JS tutoring.

* Integrating an unfamiliar authentication provider with a web app. It gave me boilerplate code that needed like 1-2 tweaks to work perfectly.

* Long winded architectural discussion giving me ideas on where to take internal libraries.

Other than that, I've actually found that the github copilot CLI is a fantastic tool I've been turning to more and more. There are so many CLI tools that I kinda sorta remember the flags for, and now I no longer have to munge about looking things up. Just earlier I needed some test api key and I just said "!! generate a random api key", and it immediately came back with "openssl rand -base64 32". That works, and saved me time googling it.

Also, your prompt here is pretty weak. For something complex you need to be very specific and give it explicit instructions on how to reason through the problem, explain it's thinking, etc. I find the more context I provide in the prompt, the better it performs. In particular leveraging the system prompt makes a big difference.

Skill issue—use it for smaller functions or learn how to write better prompts, lots of resources.

Reply on news.ycombinator.com